Disk array apparatus

ABSTRACT

A disk array apparatus using an SAS can transfer data without lowering a transfer efficiency of data even if rates of a plurality of physical links connected to a controller and storage device are different. A plurality of HDDs are connected to a controller through an expander. Data are transferred from the controller to the expander and then to HDD. In this connection, the controller and the expander transfers a set of transfer data in a plurality of the HDD-side physical links. The controller-side physical link integrates the transfer data, and multiplexes them to transfer. A plurality of HDDs-side physical links separates the transfer data to transfer in parallel.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority from Japanese Patent ApplicationNo. JP 2004-254522 filed on Sep. 1, 2004, the content of which is herebyincorporated by reference into this application.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to a disk array apparatus (storageapparatus), particularly relates to the technique for transferring databetween a controlling device (controller) and storage devices in a diskarray apparatus, and the technique employing SAS (serial Attached SCSI)as an interface with the storage devices.

BACKGROUND OF THE INVENTION

A disk array apparatus is provided as a system for realizing, forexample, reducing the risk of losing all data by storing user data in astorage area which is provided by a storage device such as a hard diskdrive (HDD). The disk array apparatus has a controller which controlsstoring data, and storage devices connected thereto. The controllercontrols storing data in a storage area based on the instruction from ahost. Also, the disk array apparatus performs RAID control employing aplurality of storage devices, and various types of control such as datareplication and backup.

Meanwhile, a SAS is provided as an interface between a computer andstorage devices. The SAS system has a computer as a device forperforming data transfer which serves as a transfer source, an enddevice such as storage devices which serve as transfer destination(target), and an expander device which relays the data transfer betweenthe end devices. A number of end devices can be connected to theexpander device. A predetermined data-transfer speed (rate) is ensuredat the physical links between the physical ports provided at eachdevice. In the connection and the data path between the end devices viathe expander device, data transfer is performed at the connection ratewhich is determined at the plurality of physical links by rate matchingor the like.

In a SAS system, there performed a process for inserting ALIGN primitiveto the transfer data in order to, for example, perform the rate matchingin a connection including physical links of different rates. The ALIGNinsertion in a SAS is described in a non patent document: Working DraftAmerican National Standard, Project T10/1562-D Revision 5, “Informationtechnology-Serial Attached SCSI (SAS)”, 4.3.2 Transmit data path, pp.45-49, 7.2.5.2 ALIGN, pp. 152-153, 7.13 Rate matching, pp. 191-193,(online), Jul. 9, 2003, (searched on Jul. 22, 2004), the Internet <URL:http://www.t10.org/drafts.htm/sas-r05.pdf>.

SUMMARY OF THE INVENTION

In order to improve performance, etc., application of SAS to a diskarray apparatus is assumed as an interface for data transfer between acontroller and storage devices. When SAS is simply applied to the diskarray apparatus, the above described ALIGN primitive is inserted inaccordance with the SAS standard in a case where, for example, aplurality of physical links have different rates.

However, the demerits of the ALIGN primitive insertion resides in that,when data transfer is performed in a connection including a slow SAS enddevice (for example, an HDD corresponding to 1.5 Gbps), the connectionrate is set to slow (for example, 1.5 Gbps) as a result of the abovedescribed rate matching or the like even if the physical link rate isfast (for example, 3.0 Gbps). Therefore, the data transfer efficiencyand bus efficiency in the connection and the data paths are lowered.

The present invention has been accomplished in consideration of theabove described problems, and an object thereof is to form an disk arrayapparatus to which SAS is applied as an interface with storage devices,and to provide techniques which enable efficient data transfer withoutlowering the data transfer efficiency and the bus efficiency even when aplurality of physical links in the connection between the controller andstorage devices have different rates.

Brief explanation of the general outline of a typical invention amongthe inventions disclosed in the present application is as the following.In order to accomplish the above described object, a disk arrayapparatus of the present invention has a plurality of storage devicessuch as HDD; and a controller (controlling device) which performscontrolling of data storing in the storage area provided by the abovedescribed storage device, in accordance with an instruction given from adata processing device which serves as a host; and the disk arrayapparatus is characterized by having the below-described technicalmeans.

(1) A disk array apparatus of the present invention has a configurationto which SAS is applied as an interface between a controller and storagedevices wherein at least one SAS expander device (expander) is connectedto the controller and the storage devices via physical links, andcommunication of data transfer according to the SAS or SATA (serial ATA)protocol is performed in the connection between the controller, theexpander, and the storage devices. Each of the controller, expander, andthe storage devices has means (for example, a circuit for processingcommunications according to the SAS protocol) for performing datatransfer in accordance with the SAS protocol. The physical link isformed by having physical ports which is provided at each member, andport line (bus) which connects between the members. The controller andthe storage devices serve as the SAS end devices. Particularly, theexpander is equipped with at least one physical port at thecontroller-side and a plurality of physical ports at the storagedevice-side. Data transfer speed (rate) is ensured at each of thephysical links for the above described data transfer. The presentapparatus is equipped with means for performing multiplex transfer inwhich, as for input/output data transferred via the plurality ofphysical links between the expander and the plurality of storage devicesin the data transfer in the connection between the controller and thestorage devices and in the data path thereof, the input/output data aremultiplexed and transferred at the controller-side (between thecontroller and the expander) physical link. By the means for performingmultiplex transfer, the controller multiplexes and transfers thetransfer data to the expander via the controller-side physical link, andthe expander transfers the transferred data in parallel to the storagedevices via the plurality of storage-side (between the expander and thestorage devices) physical links. Herein, the process is performedwithout insertion of ALIGN primitive or the like.

In addition, as means for performing the multiplex transfer, each of thecontroller and the expander are equipped with a data processing meansfor integration of multiple pieces of data into multiplex data and forseparation of the multiplex data into multiple pieces of data, and amemory for storing therein the object data to be processed. As the dataprocessing means, the controller is equipped with a first means (dataseparation/integration circuit) for performing separation/integrationprocess of the transfer data for the multiplex transfer, and theexpander is equipped with a second means (data separation/integrationcircuit) for performing separation/integration process of the transferdata for the multiplex transfer.

By the means for performing multiplex transfer, in the data transfer inthe connection between the controller and the storage devices, upon awrite process of data to the storage device, the controller integratesthe write data (the objective data to be written in the storage devices)by the first circuit and transmits, as multiplex data, to the physicalport of the expander via the controller-side physical link. Then, theexpander separates the received data by the second circuit, and theexpander transmits the separated data to the physical ports of theplurality of the storage devices via the plurality of storagedevice-side physical links and distributes the data over the storagedevices so as to perform write process. Upon read process of data fromthe storage devices, the expander transfers the read data (the objectivedata to be read from the storage devices) from the plurality of storagedevices through the corresponding physical ports and via the pluralityof storage device-side physical links. Then, the expander integrates thedata which have been received from the storage devices by the secondcircuit and transmit the data as multiplex data to the physical port ofthe controller via the controller-side physical link. Then, thecontroller performs a process of separating the data which have beenreceived from the expander, by the first circuit.

The present apparatus performs data transfer by the means for performingmultiplex transfer without insertion of ALIGN primitive or the like,even when, particularly, the rates of the physical links of thecontroller-side and storage device-side in relation to the expander aredifferent. By the means for performing multiplex transfer, in theconnection between the controller and the storage devices, and in aconfiguration where the rate of the storage device-side physical linksis slower than the rate of the controller-side physical link, themultiplex transfer is performed by employing a set of data transferredvia the plurality of slow rated physical links as the object.

In the present apparatus, between the controller and the storagedevices, there selectively performed a particular operation such as themultiplex transfer with the mediation of the process of the expander, ora normal access in which data transfer, etc. is performed directlywithout the process of the expander. When the multiplex transfer isexecuted, for example, the controller issues a command to the expanderfor the instruction, and specifies in the command, as the destinationthereof, the address of the expander and the physical ports which areemployed as the process objects. The expander has means for interpretingthe command given from the controller-side, and mediates the databetween the controller and the target storage devices by converting theframe address so as to process the multiplex transfer. The expanderrecognizes the addresses of the process-object storage devices byreferencing to the address table which has been created and retained inthe device itself. The expander transmits the SAS/SATA command and datato the storage devices by the specified physical ports corresponding tothe above described physical port specification. The expander replicatesthe command which has been given from the controller-side, and employthe command which are to be transmitting to the storage devices. Forexample, in the above described command, attribute of the processrelating to the particular process such as the degree of multiplex,e.g., duplex(2×) transfer and multiplex-4(4×) transfer, is specified.

In the present apparatus, as the multiplex transfer, double(2×),triple(3×), quadruple(4×), or multiplexing more than that are executedin accordance with, for example, the state of the storage devices (e.g.,whether connected or not, and physical link rates) and the type of theobjective data to be processed. Particularly, as the multiplex transfer,2× transfer or 4× transfer is performed by employing the transfer dataas the objects for the plurality of storage devices. In a case of 2×transfer, the controller multiplexes the transfer data which are for,e.g., two storage devices and transfers the data to the expander via thecontroller-side physical link, and the expander transfers the transferdata in parallel to the two storage devices via the storage device-sidetwo physical links. Similarly in a case of 4× transfer, the controllermultiplexes the transfer data which are for, e.g., four storage devices,and transfers the data to the expander via the controller-side physicallink, and the expander transfers the transfer data in parallel to thefour storage devices via the four storage device-side physical links.

In the present apparatus, as the plurality of storage devices connectedto the expander, for example, HDDs corresponding to SAS (SAS-HDD), andHDDs corresponding to SATA (SATA-HDD) can be employed.

Another disk array apparatus of the present invention further has thefollowing characteristics in addition to the above describedconfiguration. By the means for performing multiplex transfer, thecontroller transfers the objective data to be processed withoutmodification via the controller-side physical link. That is, one or morepath of data are integrated in order and serially transferred. Then, theexpander performs distribution of data in a predetermined size, forexample, per word units, to the plurality of storage devices via theplurality of storage device-side physical links. That is, the expanderseparates the multiplex data so as to transfer the data to the pluralityof storage devices, and transfers the separated data in parallel overthe storage devices.

In the present apparatus, as the hardware configuration, the expander ismounted, for example, on one or more disk controlling unit which isprovided at one or more chassis that forming the present apparatus, oron a board (e.g., power supply controller board) to which the diskcontrolling unit is mounted. The disk controlling unit means a dataprocessing device's request which executes input/output such asread/write to the storage devices. When a plurality of expanders isprovided in the configuration, the configuration is made such that theexpanders are connected via the physical links and can transfer datamutually. Even when the target storage devices are spread over theplurality of expanders, the multiplex transfer is performed via thephysical links between the plurality of expanders.

In the present apparatus, upon employment of the multiplex transfer, aHDDs set is formed by a plurality of physical storage devices based onthe configuration relating to the physical link rates, said a virtualstorage device (set) is created by the one set of storage devices, and apredetermined RAID group and logical unit, etc. are set over the one ormore virtual storage devices. By virtue of the setting, the device canbe applied to various types of RAID levels while employing the multiplextransfer.

The present apparatus employs the multiplex transfer to an internal datacopy operation in which the data processing device serving as the hostdoes not mediates. Upon execution of internal data copy, the controllerreads the copy source data from the storage device by use of themultiplex transfer, and performs a process of writing in the storagedevices which are the copy destination, by use of the multiplextransfer.

In another disk array apparatus of the present invention, in addition tothe above described configuration, further, the controller performsaligning of the transfer data in accordance with the parallel transfer(distribution) of data via the plurality of storage device-side physicallinks, and transfers the aligned data via the controller-side physicallink. The expander distributes the transfer data according to thealignment order, via the plurality of storage device-side physicallinks.

Another disk array apparatus of the present invention further has, inaddition to the above described configuration, a means for recognizing,by the expander, the storage device states including whether the storagedevices are connected or not and the data transfer speed, and reportingthose information to the controller. In accordance with the recognitionof the state of the storage devices, the controller and the expanderdetermines the attribute of the process including the storage deviceswhich are employed as the objects in relation to the particularoperation such as multiplex transfer.

The multiplex transfer described in above (1) as a particular operation,is expanded as described below, in addition to the simple multiplexingof data which are input to or output from the plurality of storagedevices. In the below described decompression, the transfer process ofdata is also performed by use of the plurality of storage device-sidephysical links corresponding to the controller-side physical link,therefore, effects of efficiency enhancement is attained as well as thecase of the above described simple multiplexing.

(2) Another disk array apparatus of the present invention further hasthe following characteristics in addition to the configuration describedin above (1). The present apparatus has means for performing multiplextransfer accompanied with parity process, wherein, in the data transferin the connection between the controller and the storage devices, aparity process (e.g., insertion or removal of parity) is performed withthe transfer data. By the means, the controller multiplexes the transferdata and transfers to the expander via the controller-side physicallink, and the expander distributes the transfer data, which haveundergone parity process, into data and parity and transfers them inparallel to the plurality of storage devices via the plurality ofstorage device-side physical links. In the parity process, for example,the parity data is inserted at a predetermined intervals in the datasequence of the transfer.

By the means for performing the multiplex transfer accompanied with theparity process, for example, the controller performs the parity processon the transfer data, and the data with parity are multiplexed andtransferred via the controller-side physical links. Then, the expanderdistributes the data with parity divided into data and parity andtransfers them in parallel to the plurality of storage devices via thestorage device-side physical links.

By the means for performing multiplex transfer accompanied with theparity process, for example, the controller transmits the transfer datavia the controller-side physical link without modification, the expanderperforms the parity process on the transfer data, and the data aredistributed into data and parity and transferred in parallel via theplurality of storage device-side physical links. Meanwhile, upon readprocess, when an error is present in the data read from the storagedevice, the expander performs, as the parity process, automatic datarecovery process by use of parity. In the process, check of the errorand recovery to original data are performed by XOR operations employingthe data and the parity. The controller performs, on the recovered dataobtained by the data recovery process, a process of transmitting thedata to data processing device which serves as the host (in response toa read request), or a process of writing the data to another storagedevice such as an replaced HDD or a spare HDD (process relating to copyback).

In another disk array apparatus of the present invention, the expanderis equipped with, when the means for performing multiplex transferaccompanied with the parity process is provided, a means for reportingerror information in the data transfer to the storage devices,information regarding the automatic data recovery at the expander (e.g.,information notifying that the data has been recovered by use of theparity or the later-described multiplex writing), to the controller. Thecontroller recognizes and judges the state of the error in the storagedevice based on the report of the error information and the informationregarding the data recovery.

(3) Another disk array apparatus of the present invention has aplurality of storage devices, a controller for controlling storing datain the storage devices, and an expander for connecting the storagedevices and the controller via physical links; wherein the controller,the expander, and the storage devices have means for performing datatransfer in the connection between the controller and the storagedevices in accordance with the SAS protocol, and have means forperforming, in data transfer in the connection between the controllerand the storage devices, multiplex writing in which the data same as thetransfer data is written to the plurality of storage devices. In thepresent apparatus, a set of the plurality of storage devices areemployed as the object of the multiplex writing of the transfer data,the controller transfers the objective data of the multiplex writing tothe expander via the controller-side physical link; and with theobjective transfer data of the multiplex writing, upon write process,the expander replicates and transfers in parallel the data via theplurality of physical links corresponding to the storage device which isemployed as the object of the multiplex writing so as to perform write,and upon read process, the data are read in parallel and original dataare obtained.

As the multiplex writing process, particularly, double writing processin which identical data are written to two storage devices is effective.In this case, a set of the plurality of storage devices are employed asthe object of the double writing of the transfer data. The data employedas the object of the double writing are transferred via thecontroller-side physical link, and, with the transfer data employed asthe object of the double writing, upon write process, the expanderreplicates and transfers in parallel the data via the plurality ofphysical links corresponding to the storage devices employed as theobjects of the double writing so as to perform write, and upon readprocess, the data are read in parallel and original data are obtained.

In another disk array apparatus of the present invention, thereperformed a particular operation in which the multiplex transferdescribed in above (1) and the above described multiplex writing areperformed in combination. In the present apparatus, for example, a setof the plurality of storage devices are employed as the object of thedata distribution, and a pair of the storage devices are employed as theobject of the double writing of the each of the separated pieces of dataof the above described data distribution. The objective data of the datadistribution and the double writing are transferred via thecontroller-side physical link, and with the transfer data, the expanderperforms the data distribution and the double writing via the pluralityof physical links corresponding to the objective storage devices.

In another disk array apparatus of the present invention, thereperformed a particular process in which the multiplex transferaccompanied with the parity process described in above (2) and themultiplex writing are performed in combination. In the presentapparatus, a set of a plurality of storage devices is employed as theobject of the data distribution, the pieces of separated data of thedata separation are employed as the object of the parity process, and apair of the storage devices are employed as the object of the doublewriting of each piece of the separated data including parity which isgenerated in the parity process. The objective data of the datadistribution, the parity process, and the double writing are transferredvia the controller-side physical link, and with the transfer data, theexpander performs the data distribution, the parity process, and thedouble writing via the plurality of physical links corresponding to theobjective storage devices.

(4) Another disk array apparatus of the present invention further hasthe below described characteristics in addition to the configurationdescribed in above (1). The present apparatus has means for performing,in data transfer performed in the connection between the controller andthe storage devices, compression/decompression process of transfer data,and multiplex transfer accompanied with compression/decompression. Thecontroller and the expander have means for performingcompression/decompression process of transfer data, the compressed dataof the transfer data are transferred via the controller-side physicallink, and the decompressed data of the compressed data are transferredvia the plurality of storage device-side physical links.

(5) Another disk array apparatus of the present invention ischaracterized by having a configuration, as another configurationrelating to the application of the expander, in which an expander isconnected to a controller and storage devices via physical links, and a“data separation/integration end device” is connected to the expanderfrom outside via bus or the like. The expander performs, except theprocesses relating to the particular operation such as multiplextransfer described in above (1), communication processes according tothe SAS protocol. The data separation/integration end device has meansfor performing, as a process relating to the particular operation suchas the multiplex transfer, particularly, a data separation/integrationprocess, that is, a process of integrating a plural pieces of data intomultiplex data and separating the multiplex data into a plural pieces ofdata. The data separation/integration end device has, in the connectionto the expander, a path for communicating with controller-side, and aplurality of paths for communicating with the storage device-side. Uponthe execution of the particular process, the data transfer communicationis performed in the connection between the controller and the storagedevices with the mediation of the process at the dataseparation/integration end device. The communication for dataseparation/integration process is performed between the expander and thedata separation/integration end device.

Among the inventions disclosed in the present specification, the effectsattained by a typical invention are briefly explained as below.

According to a disk array apparatus of the present invention, thereformed a disk array apparatus to which SAS is applied as an interfacewith storage devices, and there enabled efficient data transfer withoutlowering the data transfer efficiency and the bus efficiency even when aplurality of physical links in the connection between the controller andstorage devices have different rates.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1A is the drawing showing the external appearance of the hardwareconfiguration of a disk array apparatus of an embodiment which iscommonly employed in disk array apparatus of embodiments of the presentinvention;

FIG. 1B is the drawing showing the external appearance of the hardwareconfiguration of a disk array apparatus of an embodiment which iscommonly employed in disk array apparatus of embodiments of the presentinvention;

FIG. 2 is a functional block diagram of the entire system configurationrelating to the disk array apparatus of the embodiment;

FIG. 3 is a functional block diagram of entirety of another systemconfiguration relating to the disk array apparatus of the embodiment;

FIG. 4 is a more detailed functional block diagram of the connectionbetween a controller and an expander in the disk array apparatus of theembodiment;

FIG. 5 is an explanatory diagram illustrating the general outline of atypical process of particular processes performed in a disk arrayapparatus in the embodiments of the present invention;

FIG. 6 includes diagrams showing an SSP frame of SAS and a SAS addressformat;

FIG. 7 is a diagram showing a format example of a particular commandwhich is employed in the embodiments of the present invention;

FIG. 8 is an explanatory diagram representing a process model of apreceding art of the present invention, wherein ALIGN primitive isinserted in accordance with the SAS standard in a configuration of adisk array apparatus to which SAS is simply applied;

FIG. 9 is an explanatory diagram representing a model of a particularprocess (multiplex transfer) in the disk array apparatus of a firstembodiment of the present invention;

FIG. 10A is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a second embodiment of the presentinvention;

FIG. 10B is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a second embodiment of the presentinvention;

FIG. 11A is an explanatory diagrams representing a setting in a case inwhich the particular operation of the second embodiment of the presentinvention is applied in an actual RAID system;

FIG. 11B is an explanatory diagrams representing a setting in a case inwhich the particular operation of the second embodiment of the presentinvention is applied in an actual RAID system;

FIG. 12 is an explanatory diagram representing a process of a case inwhich the particular operation of the second embodiment of the presentinvention is applied to a data copy operation performed in the diskarray apparatus;

FIG. 13A is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a third embodiment of the presentinvention;

FIG. 13B is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a third embodiment of the presentinvention;

FIG. 14A represents an example of processing procedure in a case inwhich the particular operation of the third embodiment of the presentinvention is applied to a RAID system;

FIG. 14B represents an example of processing procedure in a case inwhich the particular operation of the third embodiment of the presentinvention is applied to a RAID system;

FIG. 15 shows a setting screen for RAID groups which correspond to theparticular operation of the third embodiment;

FIG. 16A is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a fourth embodiment of the presentinvention;

FIG. 16B is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a fourth embodiment of the presentinvention;

FIG. 17A is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a fifth embodiment of the presentinvention;

FIG. 17B is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a fifth embodiment of the presentinvention;

FIG. 18 is an explanatory diagram of automatic data recovery employingparity, and data recovery to a spare HDD, etc. upon read, in relation toa particular process performed in the disk array apparatus of the fifthembodiment of the present invention;

FIG. 19A is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a sixth embodiment of the presentinvention;

FIG. 19B is an explanatory diagrams representing a model of a particularprocess in a disk array apparatus of a sixth embodiment of the presentinvention;

FIG. 20 is an explanatory diagram of automatic data recovery employingmultiplex writing and data recovery to a spare HDD, etc. upon read, inrelation to a particular process performed in the disk array apparatusof the sixth embodiment of the present invention;

FIG. 21A is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of a seventh embodiment of the presentinvention;

FIG. 21B is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of a seventh embodiment of the presentinvention;

FIG. 22 is an explanatory diagram of automatic data recovery and datarecovery to a spare HDD, etc. upon read, in relation to a particularprocess performed in the disk array apparatus of the seventh embodimentof the present invention;

FIG. 23A is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of an eighth embodiment of the presentinvention;

FIG. 23B is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of an eighth embodiment of the presentinvention;

FIG. 24A is an explanatory diagram of automatic data recovery and datarecovery to a spare HDD, etc. upon read, in relation to a particularprocess performed in the disk array apparatus of the eighth embodimentof the present invention;

FIG. 24B is an explanatory diagram of automatic data recovery and datarecovery to a spare HDD, etc. upon read, in relation to a particularprocess performed in the disk array apparatus of the eighth embodimentof the present invention;

FIG. 25A is an explanatory diagram representing a model of a processperformed by an HDD information reporting function which is provided ina disk array apparatus of a ninth embodiment of the present invention;

FIG. 25B is an explanatory diagram representing a model of a processperformed by an HDD information reporting function which is provided ina disk array apparatus of a ninth embodiment of the present invention;

FIG. 26 is a table showing, in the ninth embodiment of the presentinvention, regarding the combination of the state of two HDDs, therelation between availability of automatic data recovery by an expander,corresponding embodiments, and operations executed by the expander;

FIG. 27A is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of a tenth embodiment of the presentinvention;

FIG. 27B is an explanatory diagram representing a model of a particularprocess in a disk array apparatus of a tenth embodiment of the presentinvention;

FIG. 28 is a block diagram representing a configuration of a disk arrayapparatus of an eleventh embodiment of the present invention;

FIG. 29 is a flow chart showing a procedure in a case where, as anoperation of the expander, data are transferred to HDDs based on thecommand from the controller in accordance with the process performed inthe disk array apparatus of the embodiments of the present invention;and

FIG. 30 is a flow chart showing a procedure in a case where, as anoperation of the expander, data are transferred from HDDs based on thecommand from the controller in accordance with the process performed inthe disk array apparatus of the embodiments of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will next be described in detailbased on drawings.

First Embodiment

A disk array apparatus of a first embodiment of the present inventionwill be described. A hardware configuration commonly employed inembodiments of the present invention will be explained first, and thencharacteristic processes, etc. which are performed on the hardware willbe explained.

<Hardware Configuration>

FIGS. 1A and 1B are the drawings showing the external appearance of thehardware configuration of a disk array apparatus which is commonlyemployed in disk array apparatus of embodiments of the presentinvention. FIG. 1A shows the front of the device, and FIG. 1B shows therear of the device. The present disk array apparatus 100 has aconfiguration in which a rack frame 111 serves as a base, mount frames112 are formed over a plurality of blocks in the vertical directioninside the rack frame 111, and a basic chassis 120 (disk-array-controlhousing) and additional chassis 130 (HDD housing) are attached along themount frames 112 in a manner that they can be pulled out. In the presentapparatus 100, one basic chassis 120 is attached to the lowest block,and a plurality of additional chassis 130 can be attached to upperblocks. Each of the chassis is equipped with boards (circuit boards) andunits for providing various functions of the present apparatus 100. Abasic chassis 120 is a chassis for housing therein a controller board59, etc. for forming a controller 10 of the disk array apparatus. Theadditional chassis 130 is a chassis for housing HDDs 30, and may beadded in accordance with needs.

In the configuration of the present apparatus 100, an expander (20),which is described later, is applied to each of the connection unitbetween the basic chassis 120 and the additional chassis 130 and theconnection unit between additional chassis 130, that is, the expander(20) is applied to the unit of a power supply controller board (56). Adisk array apparatus having scalability can be formed by employingexpanders in the above described manner.

In the front of the device, there provided space to which a plurality ofunits of the basic chassis 120 and the additional chassis 130 loadedwith HDDs 30 can be attached in a row. The HDDs 30 can be attached anddetached in respective attached positions. In addition, in the front ofthe basic chassis, a battery unit serving as a backup power supply, adisplay panel for displaying the state of the device, a flexible diskdrive for loading programs, etc. are provided.

In the rear of the device, power supply controller boards 56 and powersupply units, etc. are provided on the basic chassis 120 and theadditional chassis 130. In the rear of the basic chassis, controllerboards 59, a cooling fan unit, etc. are provided.

A backboard is provided in each of the chassis for connecting themembers, and each of the boards, units, a plurality of HDDs 30, etc. areconnected to the backboard. The members communicate with one another viathe wirings of the backboard.

The controller board 59 controls data storing to the HDDs 30 based onthe instructions from a data processing device 300 which serves as ahost. On the controller board 59, an interface for communicating withthe host, a cache memory, a shared memory, an interface forcommunicating with HDDs 30, a circuit having a function such as for thecontrol by a RAID system and for monitoring the state of HDDs 30, etc.are mounted. The functions such as communication interface and cachememory may be mounted on another board which is separated from thecontroller board. In the configuration, two controller boards 59 areredundantly attached in order to ensure the security regarding thecontrol of the HDDs 30 in the basic chassis 120.

In the interface provided in the controller for communicating with thehost, a SAN (Storage Area Network) formed by the Fibre Channel protocol,a LAN (Local Area Network) formed by a protocol such as Ethernet(registered trademark), or a connection adopting a predeterminedstandard such as SCSI is provided as an external connector for theconnection with the host. The disk array apparatus is connected with thedata processing device 300 via a communication cable connected to theexternal connector.

The power supply controller board 56 connects the chassis and performs,for example, control of a system such as for power supply among chassisand control of HDDs 30. External SAS cables 91 are connected to theconnectors provided at the power supply controller boards 56, and thepower supply controller boards 56 are connected with one another by theexternal SAS cables 91. The power supply controller board 56 isconnected with a communication path which performs communication by abuilt-in SAS expander with a protocol such as SAS and SATA, so as to beable to communicate with the plurality of HDDs 30 in each of thechassis. On the power supply controller board 56, in addition to thecircuit forming the SAS expander, a circuit which performs, for example,monitoring of the state of an AC/DC power supply, monitoring of thestate of the HDDs 30, and control of power supply to the HDDs 30, aremounted. The various functions such as power supply controlling functionprovided in the power supply controller board 56 may be provided in thecontroller board 59.

The power supply unit is equipped with an AC/DC power supply, etc., andsupplies DC electricity to each of the members in the chassis such asthe HDDs 30 and the boards. The power supply unit is connected with thepower supply controller board 56 and supplies power to each of the HDDs30 according to the signals from the power supply controller board 56.In the configuration, two power supply controller boards 56 and twopower supply units are redundantly attached to each of the chassis inorder to ensure the security regarding the power supply to the chassis.

As the HDD 30 which is attached and connected to the chassis, there maybe employed a 2.5-inch magnetic disk and a 3.5-inch magnetic disk havingcommunication interfaces different from each other, in addition, havingdifferent I/O performances, power consumptions, and lengths of life. The2.5-inch magnetic disk has inferior I/O performance and shorter lifecompare with the 3.5-inch magnetic disk, however, has an advantage interms of the small power consumption.

<System Configuration (1)>

FIG. 2 is a functional block diagram of the entire system configurationrelating to the disk array apparatus 100. In the entire computer systemconfigured by including the disk array 100, the disk array apparatus 100and the data processing device 300 serving as the host thereof areconnected by an FC (Fibre Channel) interface via a SAN (Storage AreaNetwork) 200. The connection with the host side is not limited to theSAN 200 and FC, and may be connected by others. The disk array apparatus100 has a controller 10, an expander 20, HDDs 30, and connection unitssuch as a bus and physical port for connecting these members. A SASphysical link (40) between the controller 10 and the expander 20 has arate of 3.0 Gbps. SAS physical links (50) between the expander 20 andthe HDDs 30 have a rate of 1.5 Gbps.

The data processing device 300 is, for example, a personal computer usedby a user, a workstation, or a mainframe computer. The data processingdevice 300 includes a program for utilizing the disk array apparatus100, and a communication interface which corresponds to FC forcommunicating with the disk array apparatus 100. The data processingdevice 300 issues a command to the disk array apparatus 100, forexample, for performing read or write of data on the storage areaprovided at the HDDs 30. The disk array apparatus 100 processes read,write, etc. of data based on the accepted instruction given from thedata processing device 300.

The disk array apparatus 100 has a function for performing communicationin accordance with SAS in the communication connection between thecontroller 10 and the HDDs 30 via the expander 20. The controller 10 andthe HDDs 30 serve as SAS end devices. The SAS expander devices areapplied to the power supply controller boards 56 which serve asconnection units between the basic chassis 120 and the additionalchassis 130 and between a plurality of additional chassis 130. In thedisk array apparatus 100, the performance thereof is enhanced byconnecting a plurality of additional chassis 130 including expanders 20in accordance with needs.

A controller 10 is mounted on, for example, the controller board 59 inthe basic chassis 120. In the diagram, the controller 10 and the HDDs 30are separately shown in different chassis so as to be easily understood.The controller 10 has a plurality of physical ports (PHY) connected withthe expander 20-side by physical links (40).

The expander 20 mutually connects the controller 10 and the plurality ofHDDs 30 so as to relay data transfer, and performs particular operationssuch as multiplex transfer. The expander 20 is a device formed bymounting the SAS expander device and functions characteristic in thepresent embodiment on the part of the disk controlling unit of the powersupply controller board 56 in each of the chassis. The diagram shows aconfiguration in which the SAS expander device is mounted on the diskcontrolling unit in each of the additional chassis 130; however theconnection configuration between the controller 10, the expander 20, andthe HDDs 30 is not limited to the present configuration. The expander 20has a plurality of physical ports (PHY) for connecting with thecontroller 10-side and another expander 20-side in another additionalchassis 130 via physical links (40), and for connecting with the HDD30-side via physical links (50). Connections by the communicationinterface based on SAS are provided between the controller 10, theexpander 20, and the HDDs 30, and between the plurality of the expanders20 so as to make them mutually communicatable.

The HDD 30 is a HDD corresponding to SAS (SAS-HDD), or a HDDcorresponding to SATA (SATA-HDD). In a case of a slow xfer ratedSAS-HDD, the xfer rate at the SAS physical links (50) between theexpander 20 and the HDDs 30 is, for example, 1.5 Gbps which is slowerthan the rate at the controller 10-side. In a case of a fast SAS-HDD,the rate at the SAS physical links (50) between the expander 20 and theHDDs 30 is, for example, 3.0 Gbps which is the same speed as the rate atthe controller 10-side. The HDDs 30 have unillustrated physical portscorresponding to the physical ports of the expander 20-side, and performread/write of data in units such as blocks or sectors on the disks basedon the command or the data received via the physical links (50).Addresses unique in the system, SAS addresses particularly in a case ofSAS-HDDs, are given to the HDDs. Meanwhile, the SAS protocol is employedin the communication with the SAS-HDDs, and the SATA protocol isemployed in the communication with the SATA-HDDs.

The SAS system also has connectivity with SATA devices, and the expander20 corresponds to any of the connections with SAS-HDDs and SATA-HDDs.The SAS protocol includes the physical layer, the link layer, the portlayer, and the transport layer. A SAS port includes the layers. Aphysical port (denoted by “Phy” in the diagram) includes the physicallayer and the link layer. The transport layer performs, for example, aprocess in which commands, data, status, etc. are encapsulated in a SASframe and assigned to the port layer. The port layer performs a packet(frame) transfer process after the physical port for transmitting thepacket (frame) is selected and the connection is established. The linklayer controls the physical layer for the connection management. Thephysical layer includes the hardware for transmitting signals to a portline (bus).

The controller 10 has a CPU 11, a memory 12, a channel controlling unit13, a data controller 14, a cache memory 15, a disk controlling unit 16,and a plurality of physical ports (Phy). The controller 10 is identifiedby a SAS address (controller address). Also, for example, the channelcontrolling unit 13 and the disk controlling unit 16 may be a plural.

The CPU 11 executes control programs by using the memory 12, andrealizes various functions of the controller 10. The channel controllingunit 13 is a communication processing unit which is connected to the SAN200 and provides a communication function (FC interface) in accordancewith the FC protocol. The channel controlling unit 13 communicates with,for example, another communication processing unit at the host-side andanother disk array apparatus. The channel controlling unit 13 isconnected to the data controller 14 and performs read/write of data onthe cache memory 15.

The data controller 14 is an LSI having a data separation/integrationcircuit 17. The data controller 14 is connected to the CPU 11, thechannel controlling unit 13, the cache memory 15, and the diskcontrolling unit 16, and performs data communication and data processingamong the members. The data controller 14 performs read/write ofprocessing data, particularly transfer data with the host, on the cachememory 15.

The cache memory 15 is employed for storing processing data such as userdata and commands, and particularly the transfer data relating tomultiplex transfer functions are temporarily retained therein. Forexample, when normal access is performed, corresponding to the datainput/output requests from the host such as read and write, the channelcontrolling unit 13 stores write data, etc. in the cache memory 15 viathe data controller 14. The disk controlling unit 16 performsinput/output processes on the cache memory 15 via the data controller 14corresponding to the commands according to the instructions from a CPU11.

The disk controlling unit 16 is connected with the data controller 14and the plurality of physical ports via bus, and performs processingdata input/output on the expander 20 and the HDDs 30. The diskcontrolling unit 16 performs read/write of data on the cache memory 15via the data controller 14. The disk controlling unit 16 has acommunication function according to SAS.

The data separation/integration circuit 17 performs dataseparation/integration processes relating to, for example, the multiplextransfer functions. The data separation/integration circuit 17 performsa process of integrating the transfer data given from the host-side inaccordance with the type of the particular operation, and separating thetransfer data given from the expander 20-side. The processes performedby the data separation/integration circuit 17 will be described later.

In the controller 10 and the expander 20, port groups are formed by aplurality of physical ports. In the example configuration, two portgroups A and B are formed from eight physical ports. In the diagram, thebusses included in the port group A are indicated by solid lines, andthe bus included in the port group B are indicated by dotted lines. Thephysical ports of the controller 10-side corresponds to the physicalports of the expander 20-side in the port groups. In the connectionbetween the controller 10 and the HDDs 30, data transfer or the like canbe performed by use of the physical ports and the port groups. When afailure occurs in a data path, another physical port may be selected forswitching. A multiplex transfer, etc. can be performed by use of oneport line (bus) between the physical ports.

The expander 20 has a data separation/integration circuit 27, inaddition to a function as a disk controlling unit for controlling theHDDs 30. The expander 20 is connected to the disk controlling unit 16 inthe controller 10-side, the HDDs 30 which are attached and connected inthe additional chassis 130, and another expander 20, each of them areconnected by the SAS physical links (40 and 50) via the physical portsand bus. For example, the expander 20 has physical ports correspondingto the two paths of port groups A and B. In the diagram, among thephysical ports provided at the expander 20 for communicating with thecontroller 10-side, four physical ports corresponding to the port groupare collectively represented by one member. In the physical links, athin line represents one port line (bus), and a bold line collectivelyrepresents four port lines. The physical links connecting betweenexpanders 20 has a rate of, for example, 3.0 Gbps as well as thecontroller 10-side. All of the plurality of HDDs 30 in the additionalchassis 130 is connected to the expander 20 via the port lines (bus) andthe two paths of port groups provided at the expander 20. Each of theHDDs 30 is connected to two physical ports corresponding to the twopaths of port groups.

The number of the physical ports and the number of connectable HDDs,etc. are not limited to that of the present configuration and may beincreased or decreased. In the embodiments, in the communicationconnections between the controller 10, the expander 20, and the HDDs 30,a combination of physical link rates of fast 3.0 Gbps (controller10-side) and slow 1.5 Gbps (HDD 30-side) is described as a basiccombination. However, the combination is not limited to this, and otherrates may also employed, for example, a combination of 6.0 Gbps and 3.0Gbps, or a combination further including 1.5 Gbps may be employed.

<System Configuration (2)>

FIG. 3 is a functional block diagram of another system configurationrelating to the disk array apparatus 100. In the present configuration,the members on the data path through the host to the HDDs 30, such asthe controller 10 and expander 20 are doubled. When, for example, afailure occurs in one path, failover which switches to the other pathand continues the process, or load balancing can be performed. Themembers provided in the controllers 10 and the expanders 20 areapproximately the same as those in the above described systemconfiguration. In the present configuration, multiplex transferfunctions, etc. can also be executed in the same manner.

Each of the controllers 10 and the expanders 20 is equipped with aplurality of physical ports and corresponds to the two paths of portgroups A and B. The physical ports are connected in combinations so asto obtain redundancy between the controller 10, the expander 20, and theHDDs 30. Each of two controllers #0 and #1 is connected in the basicchassis 120. Each of the controllers 10 is connected to the SAN 200 atthe channel controlling unit 13. In each of the controllers 10, two portgroups A and B are connected to the disk controlling unit 16. In theadditional chassis 130, two expanders #0 and #1 are connected. Each ofthe expanders 20 is connected to the both controllers 10. That is, theport group A of the controller #0 and the port group A of the controller#1 are connected to the expander #0. The port group B of the controller#0 and the port group B of the controller #1 are connected to theexpander #1. Even when connection failure occurs in one of the portgroups, processes can be continued by switching to the connection of theother port group. All of the plurality of HDDs 30 in the additionalchassis 130 is connected to each of the expander 20 via the bus andthrough the plurality of physical ports provided at the expander 20.Also, the expander 20 has two paths of physical ports for connectingwith another expander 20 in another additional chassis 130, and thesefour physical ports and these four port lines are collectivelyrepresented by one member in the diagram.

The expander 20 has a switch for switching the paths among the physicalports in the expander 20, which is switched depending on the datatransfer destination.

<Connection Between the Controller and the Expander>

FIG. 4 is a more detailed functional block diagram of the connectionbetween the controller 10 and the expander 20 in the disk arrayapparatus 100. Particularly, the configurations of the data controller14, the disk controlling unit 16, and the expander 20 is shown.

The data controller 14 in the controller 10 has a buffer 141 and a datacontrolling circuit 142. The data separation/integration circuit 17 isconfigured by the functions of the buffer 141 and the data controllingcircuit 142. The data controlling circuit 142 performs data processingsuch as data separation/integration while buffering the data in thebuffer 141.

The disk controlling unit 16 in the controller 10 has SAS protocolcontrolling units 161 corresponding to the two paths of port groups Aand B, and a plurality of physical ports. The SAS protocol controllingunits 161 are connected to the data controlling circuit 142 and the portgroups by bus, and performs processes in accordance with the SASprotocol.

The expander 20 has a data separation/integration circuit 27, a directorcircuit 28, and a plurality of physical ports. The dataseparation/integration circuit 27 has a buffer 271 and a datacontrolling circuit 272 including an XOR circuit 273. The physical portsprovided at the expander 20 are connected via the director circuit 28 bybus. The expander is connected to another controller (redundantcontroller) 10 or another expander 20 of another additional chassis 130via the physical ports of the expander 20.

The director circuit 28 switches the paths among the physical ports inthe expander 20. In a case of a normal access wherein a particularoperation such as multiplex transfer is not performed, there selected apath which directly connects the physical port at one side to thephysical port at the other side via the director circuit 28, not via thedata separation/integration circuit 27. In a case of an access in whicha particular operation such as multiplex transfer is performed, thereselected a path which connects the physical port at one side to the dataseparation/integration circuit 27, via the director circuit 28, wheredata processing is performed, and connected to the physical port at theother side.

The data separation/integration circuit 27 performs processescorresponding to the controller 10-side for the particular operations.The data controlling circuit 272 performs data processing such as dataseparation/integration while buffering the data to the buffer 271. When,for example, a process employing parity is performed, the datacontrolling circuit 272 performs the process by utilizing the XORcircuit 273.

<Particular Processes>

FIG. 5 is an explanatory diagram illustrating the general outline of atypical process of particular processes (multiplex transfer) performedin a disk array apparatus in the embodiments of the present invention.The typical process illustrated in the diagram corresponds to theprocess of the second embodiment which is specifically described later.In the diagram, particularly, a case of duplex transfer is illustrated,however further multiplexing is performed in the same manner.

The expander 20 exists in the connection between the controller 10 andthe HDDs 30 serving as the end devices, and in the data paths thereof;and the particular operations are executed through the processes at theexpander 20. In the data paths between the controller 10 and the HDDs30, a physical link 40 between the controller 10 and the expander 20,and physical links 50 (50 a and 50 b) between the expander 20 and theHDDs 30 are provided. In the present configuration, SAS-HDDs areemployed as the HDDs 30 having a slow-speed physical link rate (1.5Gbps) compare with the physical link rate (3.0 Gbps) of the controller10-side.

A multiplex transfer is performed as a particular operation by dataseparation/integration processes at the controller 10 and the expander20. In the multiplex transfer, the data to be transferred via theplurality of HDD-side physical links 50 are employed as the object, andmultiplexed and transferred via the controller-side physical link 40.Particularly in a duplex transfer, the data to be transferred via theHDD-side physical links 50 corresponding to the two HDDs 30 are duplexand transferred via the controller-side physical link 40.

In the diagram, two HDDs 30, a SAS-HDD #A (30 a) and a SAS-HDD #B (30b), are provided as an example of a set (group) of HDDs 30 which areemployed as the multiplex objects. Data A {A0, A1, . . . } to beinputted to or output from the HDD 30 a are transferred via the physicallink 50 a between the expander 20 and the HDD 30 a, for example, by aread operation from the HDD 30. In the same manner, data B {B0, B1, . .. } to be inputted to or outputted from the HDD 30 b are transferred viathe physical link 50 b between the expander 20 and the HDD 30 b. Herein,A0, B0 or the like are the data per one-word (dword) units.

The controller 10 multiplexes and transfers the data to be transferredin parallel to the two HDDs 30 a and 30 b in a predetermined data sizesuch as that in a word (dword) unit via the fast-speed side physicallink 40. In the multiplex data, the data (A and B) in a plurality ofpaths are arranged alternately per word units. For example, multiplexdata {A0, B0, A1, B1, . . . } are transferred via the physical link 40.

When write operation to the HDDs 30 is performed, the controller 10integrates the two paths of data A and B which are to be transferred tothe two HDDs 30 by the data separation/integration circuit 17, andtransfers the data as duplex data via the physical link 40. The expander20 separates the duplex data received from the controller 10-side viathe physical link 40, by the data separation/integration circuit 27, andtransfers the data in parallel via the slow-speed side two physicallinks 50.

Similarly, in the direction from the HDDs 30 to the controller 10, theexpander 20 integrates the two paths of data A and B received from theHDD 30-side via the slow-speed side two physical links 50, by the dataseparation/integration circuit 27, and transfers it as duplex data viathe fast-speed side physical link 40. The controller 10 separates theduplex data received from the expander 20-side via the fast-speed sidephysical link 40, by the data separation/integration circuit 17, andobtains the data as two paths of data A and B from the two HDDs 30.

In the present processing example, one-word unit of data are drawn fromthe head of the data A and B which are transferred at approximately thesame timing via the slow-speed side two physical links 50 a and 50 b,and two units thereof are integrated and alternately arranged. As aresult, the duplex data assumes a data sequence such as {A0, B0, A1, B1,. . . }. In the multiplex transfer, the timing for transferring data inparallel from a plurality of HDDs 30 does not have to be the completelysame timing.

The rate of the fast-speed side physical link 40 is double rate of theslow-speed side. Therefore, the data transfer process via the twophysical links 50 a and 50 b is balanced with that of the physical link40 by the duplex transfer via the physical link 40. ALIGN insertion isnot performed for controlling the rate of physical links, accordingly,the data transfer efficiency and the bus efficiency in the connectionbetween the controller 10 and the HDDs 30 and in the data paths areimproved.

<Command>

An example of a command employed for data communication orcontrol-information communication between the controller 10 and theexpander 20 will be explained. When a particular operation such asmultiplex transfer is performed, the controller 10 transmits a command(hereinafter, referred to as a particular command) according to SAS tothe expander 20, and the expander 20 interprets the accepted command andexecutes corresponding particular operations. In the embodiments, when aparticular operation is performed, an expander address is employed forthe access to the HDDs 30. The expander address is the information whichuniquely identifies the expander 20 in the system. When the particularoperation is performed, the controller 10 transmits a particular commandto the objective expander 20 by employing the expander address as thedestination address. An SSP (Serial SCSI Protocol) command according toSAS is utilized as the particular command. When the expander 20 acceptsthe particular command from the controller 10, the expander 20 executesthe particular operation when the address is addressed to the expander20. When the accepted command is addressed to another expander 20, theexpander 20 transfers the command to this another expander 20 via aphysical link. The controller 10 directly performs a normal access tothe expander 20 and the HDDs 30 in accordance with the SAS protocolwithout employing the expander address.

The expander 20 basically does not perform command conversion nor acommand processing such as that executed by the controller 10, andperforms the following conversion operations. First, the expander 20performs command replication and SAS address conversion. The commandreplication is a process for transmitting commands to HDDs 30 serving asa plurality of transfer destinations (targets), and commands to betransmitted to the HDDs 30-side are created by replicating theparticular command received from the controller 10-side. The SAS addressconversion is a process for converting the expander address in theaccepted command to SAS addresses (HDD addresses) of HDDs 30 which aretransfer destinations, while referencing an address table included inthe expander 20.

Secondly, the expander 20 performs data manipulations(replication/separation/integration/XOR) relating to the multiplextransfer, and conversion of the data lengths. The data manipulationsare, for example, replication of transfer data for transferring data tothe plurality of HDDs 30, separation of multiplex data, integration ofmultiple pieces data, and an XOR operation process for a parity process.

Thirdly, the expander 20 performs monitoring of the execution time andmanagement of error code data. The monitoring of the execution timemanages such that the required process is completed within theprocessing time limit set for each command. The management of error codedata is a process of, for example, generating an error codecorresponding to, e.g., an error which occurs in data read from the HDDs30, saving the code in a memory, and reporting to the controller10-side.

<Whole Processing Procedure>

The whole processing procedure in the disk array apparatus 100 will beexplained. Processes are performed basically in accordance with thefollowing procedures (1) to (4). The disk array apparatus 100 canselectively execute a normal access or a particular operation based onthe decision or the setting of the controller 10.

Procedure (1): Upon start-up of the disk array apparatus 100, theexpander 20 performs rate negotiations (negotiations) with each of theconnected HDDs 30 and the controller 10. As a result of the ratenegotiations, the data-transfer speeds (physical link rates) of therespective physical links (40 and 50) are assured. For example, thephysical link 40 is determined to have a rate of 3.0 Gbps, and thephysical links 50 are determined to have a rate of 1.5 Gbps. It must benoted that the rate negotiations are the processes different from ratematching for controlling rates among the physical links. By the processincluding the rate negotiations, an address table for interconnectionamong the members is created in the expander 20. The connectionconfigurations between the physical ports provided at the expander 20and the controller 10, the HDDs 30, and another expander 20 are mappedin the address table. The address table is updated in accordance withchanges in the connection configurations.

Procedure (2): The controller 10 examines the speed information of eachof the HDDs 30 by normal accesses. According to the examination, actualdata-transfer speed in each of the physical links is recognized.

Procedure (3): The controller 10 calculates the ratio of the rates ofthe controller 10-side and the HDDs 30-side, and according to thecalculated ratio, determines the attribute of the process such as thetype of the particular operation to be executed, the formation degree ofa set (group) of the HDDs 30 which are employed as the object of amultiplex transfer, and the degree of multiplex. The controller 10determines to perform a multiplexing up to the ratio of the ratescalculated as described above. For example, when the combination of thephysical link rates in the connection between the controller 10 and twoHDDs 30 is 3.0 Gbps and 1.5 Gbps, the controller determines to performduplex transfer while employing the two HDDs 30 as the objects, sincethe rate of one side is two times that of the other one. Alternatively,for example, when the rates of the controller 10-side and the HDDs30-side are at the same speed, the controller determines to perform anormal access to the HDDs 30. The object of the multiplex transfer maybe in HDD units or in data units.

Procedure (4): The controller 10 gives instructions of a particularoperation such as multiplex transfer to be executed, to the expander 20by a particular command. The particular command is issued by, forexample, processing the SAS-address specifying area in an existing SSPframe header shown in FIG. 6. The instructions of the particularoperation are written in the SAS address specifying area. The expander20 interprets the particular command and realizes various types ofoperations. In the command transmission between the expander 20 and theHDDs 30, normal SAS addresses (hashed) which have been converted by theaddress table included in the expander are employed.

FIG. 6 shows an SSP frame of SAS and a SAS address format. The SASaddress specifying area (9 bytes) in the byte fields 1 to 9 in the SSPframe header (24 bytes) has a destination SAS address area (HashedDestination SAS address Fields), a source SAS address area (HashedSource SAS address Fields), and reserve areas (Reserved Fields). The SASaddress format has 8 bytes of a SAS address, however, the SSP frame hasdata (24-bit hash) shortened by a hash process.

The particular command can be issued, for example, by setting values inthe reserved areas and employing a private SAS address which is uniqueand valid only in the present system, in the SAS address specifyingarea. For example, some of the SAS addresses (for example, 24-bit hash“000000 h”) that are not practically used in a normal access, can beemployed as a particular command based on the SAS standard. SASaddresses and hash processes are described in, for example, section4.2.2 of the above described Non-Patent Document.

FIG. 7 is a format example of the particular command which is employedin the embodiments. In this case, the SAS address specifying area in theabove described SSP command frame has been processed to be a particularcommand area. For example, the attribute of the process of theparticular operation and the physical ports are specified by use of thereserved areas in the SAS address specifying area. The particularoperation and the related processes include multiplex transfer, doublewriting, parity process, information reporting, datacompression/decompression, and the combinations thereof, those describedin the embodiments. Any of these processes can be specified by theparticular command.

The above described destination SAS address areas are in the byte fields1 to 3 in the particular command, and the areas are used for specifyingthe expander address. The source SAS address areas are in the bytefields 5 to 7. The 2-bit flag area in the byte field 4 is used forspecifying the mode or pattern of the particular operation. The 6-bitcommand area in the byte field 4 is used for specifying the operationsuch as read/write. The physical port specifying areas in the bytefields 8 to 9 are used for specifying physical ports serving as theemployed object in the physical links 50 between the expander 20 and theHDDs 30. For example, four physical port information (physical port No.1 to No. 4) can be specified by use of 4×4 bits in the format. Thephysical port information is specified by, for example, physical portnumbers or physical port areas.

The controller 10 specifies various types of particular operations byuse of the flag area and the command area. In relation to the use of theflag area, a flag example is shown in the right side of the diagram. Forexample, when the flag value is “00”, a duplex transfer process isspecified. In the same manner, mode or pattern of the process isspecified, for example, “01” specifies 4× transfer process, “10”specifies double writing process, and “11” specifies both of duplextransfer and double-writing process. For example, several combinationsof particular operations are set as patterns and used.

In relation to the use of the command area, for example, the valuethereof can specify a process such as internal data copy (secondembodiment), automatic parity generation upon write (fifth embodiment),automatic data recovery by parity upon read (fifth embodiment), datarecovery by parity toward a spare HDD (spare disk) (fifth embodiment),automatic data recovery by a mirror HDD upon read (sixth embodiment, andseventh embodiment), data recovery by the mirror HDD to the spare HDD(sixth embodiment and seventh embodiment), and inquiry of the usablephysical port combination (ninth embodiment).

<A Case in which ALIGN Primitive is Inserted>

FIG. 8 is an explanatory diagram representing, for comparison with theembodiments of the present invention, a process model of a precedingtechnology of the present invention, wherein ALIGN primitive is insertedin accordance with the SAS standard in a configuration of a disk arrayto which SAS is simply applied. The diagram shows the flow of a processand data (command and objective data to be stored) between a controller,an expander, and HDDs. Particularly, the diagram shows a case in whichdata-write is performed on the HDDs {drive A and drive B} in the diskarray apparatus corresponding to a write instruction from a dataprocessing device serving as a host.

In the preceding technology, SAS is applied to the disk array apparatusas the interface for data transfer between the controller and HDDs, theexpander is connected via physical links, and data transfer for, e.g.,read/write of data is performed in accordance with the SAS standard inthe connections between the controller, the expander, and the HDDs, andin the data paths thereof. In a conceivable case, a plurality of HDDscorresponding to SAS (SAS-HDD) is connected to the expander in aphysical link rate slower than the physical link rate between thecontroller and the expander. The configuration shown in the diagram isan example of a case in which the physical link rate between thecontroller and the expander is 3.0 Gbps, and the physical link ratebetween the expander and the HDDs is 1.5 Gbps, wherein the rate of theone side is two times rate of the other side.

In the configuration to which SAS is simply applied as described above,when the HDD-side physical link rate is slower than that of thecontroller-side physical link, ALIGN primitive is inserted in the SASupon data transfer in the connection between the controller and theHDDs. In a normal access, ALIGN primitive is inserted to transfer datain the controller-side physical link of the fast-speed-side by, forexample, rate matching between the physical links in accordance with theSAS standard.

In FIG. 8, the controller temporarily retains the data given from a hostor HDDs, in a cache memory. For example, the diagram shows a state inwhich write data {data A (data for drive A)} and data B (data for driveB)} for the two HDDs {drive A and drive B} accepted from the host istemporarily retained in the cache memory without modification.

The controller issues a command for every one of the HDDs which isemployed as the target. The controller sequentially issues a writecommand and data A to the drive A, and a write command and data B to thedrive B. Herein, when the controller-side physical link rate isdifferent from the target-HDD-side physical link rate, the controllerinserts ALIGN primitive to every one-word transfer data (for example,the above described write command and the write data), and transfers thedata via the controller-side physical link. The word (dword) is a dataprocessing unit in SAS. As a result of the ALIGN primitive insertion,the rate (connection rate) in the connection between the controller andthe HDDs is adjusted. That is, the connection rate is obtained byadjusting the rate of the controller-side physical link of thefast-speed-side to the rate of the HDD-side physical link of theslow-speed-side. For example, the rate of one side is two times rate ofthe other side in the present configuration, accordingly, ALIGNcorresponding to one word is inserted to one word of transfer data. Thecontroller transmits the write command and the write data to theexpander-side physical port. Hereinafter, the transfer-processing timecorresponding to data of one-word unit is referred to as t. In thecontroller-side physical link, the data corresponding to two wordsincluding the ALIGN primitive are transferred in a transfer-processingtime of 2 t.

The expander performs relay and delivery of the command and the data,from the controller to the HDDs. The expander sequentially receives thetransfer data via the controller-side physical link, performs addressconversion by a SAS address table included in the expander, andtransmits the transfer data (write command and write data) to each ofthe target HDDs {drive A and drive B} via physical ports correspondingto them. Herein, the expander transfers the data from which the ALIGNprimitive is removed, via the slow-speed-side physical links. In thedata transfer, processing time of 2 t is required for the data ofone-word unit. The expander sequentially transfers the write command andthe data A to the drive A, and sequentially transfers the write commandand the data B to the drive B. Each of the HDDs stores the write data inthe disk based on the received write command.

When the host reads data from the HDDs, the flow of the process of theabove described write process is reversed. That is, the expander insertsALIGN primitive to the data read from the HDDs and transfers the data tothe controller-side. The controller removes the ALIGN primitive from thedata transferred from the expander and provide the data to the host.

As described above, when ALIGN insertion is performed in the connectionbetween the controller and the HDDs, accordingly the data transferefficiency is lowered at the controller-side physical link. For example,in the present configuration, the rate of the controller-side physicallink is adjusted to that of the HDD-side and lowered from 3.0 Gbps to1.5 Gbps.

<Multiplex Transfer>

FIG. 9 is an explanatory diagram representing a model of a particularprocess (multiplex transfer) in the disk array apparatus of the firstembodiment. The diagram shows the flow of the process and the data(command and the objective data to be stored) between the controller 10,the expander 20, and the HDDs 30. The rate of the controller-sidephysical link 40 is 3.0 Gbps, and the rate of the HDD-side physicallinks 50 is 1.5 Gbps, that is, the rate of one side is two times rate ofthe other side. The diagram shows a case in which data-write isperformed on, particularly, two HDDs 30 {drive A and drive B} in thedisk array apparatus, corresponding to a write instruction given fromthe data processing device 300 which serves as a host.

In the first embodiment, as a particular operation, the transfer datafor the plurality (particularly, two) of HDDs 30 are employed as theobject and multiplexed and transferred via the controller-side physicallink 40 without inserting the above described ALIGN primitive, and thedata is distributed via the plurality of HDD-side physical links 50,that is, a parallel data transfer is performed. Particularly, the HDDs30 which are connected to the expander 20 and have the same physicallink rate between them, are employed as the object of the multiplextransfer. In the multiplex transfer of the first embodiment, data in asingle or a plurality of system(s) are simply transferred sequentiallyvia the controller-side physical link 40. In relation to the particularoperation, the controller 10 does not perform special datamanipulations, and the expander 20 performs, as a data manipulation, adata separation/integration process for distributing the data to theplurality (two) of HDDs 30. Particularly, the expander 20 separates anddistributes the data in stripe units, to the plurality (two) of HDDs 30.The diagram particularly shows a case in which the two HDDs 30 arecoupled and duplex transfer is performed by employing the couple as theobject, and this case corresponds to a RAID system wherein data aresubjected to striping (division), and the data are stored in the coupleof HDDs 30.

In FIG. 9, when the host requests write, the controller 10(particularly, the channel controlling unit 13 and the data controller14) temporarily retains the write data {data A and data B} given fromthe host, in the cache memory 15 without modification. For example, thedata A and the data B are the striping data for each of the HDDs 30.

In relation to the write data (A and B) for the two HDDs 30 which areemployed as the objects of the duplex transfer, the controller 10(particularly, the data controller 14) does not issue write commands toeach of the HDDs 30, but issues a write command as a particular commandinstructing duplex transfer, to the expander 20. The particular commandserves as a replication source for creating write commands issued toeach of the target HDDs 30. The controller 10 issues the particularcommand by specifying the expander address as the destination thereof.The controller 10 sequentially transfers the particular command, and thedata A and B to the expander 20 via the physical port of the physicallink 40. For example, when the data A consists of n word(s), thetransfer-processing time thereof is nt. Meanwhile, although the commandconsists of a plurality of words, it is shown in one-word sizeabbreviation in the diagram for simplification.

Based on the particular command given from the controller 10-side, theexpander 20 performs the particular operation in the connection betweenthe controller 10 and the HDDs 30. The expander 20 sequentially receivesthe particular command and the data A and B from the controller 10-side.Herein, the expander 20 performs buffering of the particular command andthe data A and B with the buffer. Through the process in the dataseparation/integration circuit 27, the expander 20 replicates theparticular command, performs conversion of the addresses such as thedestinations by use of the address table which is included in theexpander 20, and transmits the write command and write data to each ofthe target HDDs 30. The expander 20 replicates the particular command,and changes the destination address thereof into the SAS addresses ofthe target HDDs by the above described address conversion, as a result,the write commands corresponding to the HDDs 30 are created. Theexpander 20 transfers the write commands and the write data to the HDDs30 via the two HDD-side physical links 50. In the HDD-side physicallinks 50, the transfer-processing time of 2 t is required for one-wordunit of data. Each of the HDDs 30 stores the write data in the diskbased on the write command received via the HDD-side physical links 50.

When the host reads data from the HDDs 30, the flow of the process uponthe above described write is reversed. That is, the expander 20 readsthe data, in parallel and in stripe unit, from the two HDDs 30 which arethe objects of duplex transfer, integrates the read data through aprocess in the data separation/integration circuit 27, and transfers thedata as multiplex data via the controller-side physical link 40. Thecontroller 10 separates the transfer data given from the expander 20, bythe data separation/integration circuit 17, and provides the data to thehost.

As shown in the diagram, in the above described particular operation,when the expander 20 transmits the commands and the data to each of theHDDs 30, the expander can execute transferring the command to drive B atthe same time as transferring command to the drive A, by virtue of theabove described particular command replication. However, in the expander20, the data B does not reach thereto from the controller 10-side untilthe time (nt) passes and the transfer of the data A to the drive A iscompleted. Therefore, the execution of the data transfer process to thedrive B is kept waiting.

In the present configuration, since the rate of the controller 10-sideis faster than that of the HDDs 30-side, a buffer having relativelylarge capacity for temporarily storing data in the expander 20, isrequired. However, the time occupying the bus in the controller-sidephysical link 40 becomes ½ (half) of that of the preceding art in asimple comparison, therefore, the traffic is reduced and the processingefficiency is improved. This happens not only upon write but also uponread. The traffic can be reduced in a case where the controller 10executes the data verifying or a read-and-throw-away operation with thedisks, at the same time as the corresponding operation which isperformed in accordance with the requests by a command of the host.Therefore, the execution performance of the host requests can besignificantly improved compare with the preceding art. The abovedescribed read-and-throw-away operation is an operation in which thecontroller 10 reads and checks data from the HDDs 30, and does notdeliver the data to the host.

Second Embodiment

Next, FIGS. 10A and 10B are explanatory diagrams representing a model ofa particular process in a disk array apparatus of a second embodiment.FIG. 10A shows the flow of the process and the data between thecontroller 10, the expander 20, and the HDDs 30. FIG. 10B shows therelation between input/output data and time in the controller 10 and theHDDs 30. The rate of the controller-side physical link 40 is 3.0 Gbps,and the rate of the HDD-side physical links 50 is 1.5 Gbps, thereforethe rate of one side is two times rate of the other side. The diagramshows a case in which data-write is performed on the HDDs 30 {drive Aand drive B} in the disk array apparatus, corresponding to a writeinstruction given from a host.

The second embodiment is a modification of the first embodiment wherein,as a particular operation, there performed multiplex transfercorresponding to the RAID system which employs the plurality(particularly, two) of HDDs 30 in a set. The transfer data aremultiplexed and transferred via the controller-side physical link 40.The plurality of HDDs 30 which are connected to the expander 20 and havethe same physical link rate between them, are employed as the object ofthe multiplex transfer, and the expander 20 distributes the data to theplurality of HDD-side physical links 50 in one-word units. In themultiplex transfer of the second embodiment, the data is simplytransmitted sequentially via the controller-side physical link 40. Inrelation to the particular operation, the controller 10 does not performspecial data manipulations, and the expander 20 performs, as a datamanipulation, data distribution to the plurality (two) of HDDs 30 inword units. The diagram particularly shows a case in which duplextransfer is performed, and a process corresponding to the RAID systemwherein the two HDDs 30 at the slow-speed-side are coupled and the dataare distributed and stored.

In FIG. 10A, upon write request, the controller 10 temporarily retains,in the cache memory 15, the write data given from the host withoutmodification. The write data are alternately distributed and stored inone-word units in the set of HDDs 30 which serves as the writedestination, and the data are not particularly limited as the databelonging to either one of the HDDs 30.

In relation to the write data which are employed as the objects of themultiplexing (doubling), the controller 10 issues a particular commandinstructing multiplex transfer, to the expander 20. The particularcommand serves as a replication source for creating write commands whichare to be delivered to the plurality of HDDs 30. The controller 10issues the particular command by specifying the expander address as thedestination thereof. The controller 10 specifies, in the physical portareas in the particular command, the physical port number, etc. of thoseemployed as the objects of the multiplex transfer. The controller 10sequentially transfers the particular command, and the multiplex data tothe expander 20 via the physical link 40 of the fast-side. In thediagram, the multiplex data are represented by a data sequenceconsisting of one-word units {A, B, C, D, E, F, G, H, . . . , X, Y, . .. }.

The expander 20 receives the particular command and the multiplex datasequentially from the controller 10-side, and separates the multiplexdata by the data separation/integration circuit 27. The expander 20replicates the particular command, performs address conversion by use ofthe address table, and delivers the write command and the write datawhich have been separated in one-word units, to each of the HDDs 30which are employed as the targets. The expander 20 distributes the datavia the specified physical ports, in accordance with the physical portinformation included in the particular command. The expander 20transfers the write commands and the separated write data to the HDDs 30via the plurality (two) of slow-side physical links 50. For example, thecommand data and the write data {A, C, . . . } are sequentiallytransmitted to the drive A, and the command data and the write data {B,D, . . . } are sequentially transmitted to the drive B. Each of the HDDs30 stores the separated write data in the disk based on the receivedwrite command.

When the host reads data from the HDDs 30, the flow of the process uponthe above described write is reversed. That is, the expander 20 readsthe data, in word units, from the plurality (two) of HDDs 30 which arethe objects of multiplex transfer, integrates the read data, andtransfers the data as multiplex data to the controller 10-side. Thecontroller 10 separates the transfer data given from the expander 20,and provides the data for the host.

As shown in FIG. 10B, in the relation between the input/output data andtime, when the data in word units {A, B, C, D, . . . } are sequentiallytransmitted from the controller 10-side to the expander 20, the data {A,C, E, G, . . . } are correspondingly stored in the drive A and the data{B, D, F, H, . . . } are correspondingly stored in the drive B both atthe substantially same timing except the transfer delay time t.

The second embodiment is different from the first embodiment in that theprocess waiting time (above described nt) accompanied with the datatransfer to the HDDs 30 are not necessary, so that the efficiency isimproved. There attained a relation wherein “the data transfer speed ofthe controller side (3.0 Gbps)=the data transfer speed of the drive Aside (1.5 Gbps)+the data transfer speed of the drive B side (1.5 Gbps)”,and a buffer for controlling the speed is not required, in principle.Therefore, the configuration can be formed with the minimum componentsrequired, which is advantageous. When the rates in the side of theplurality of HDDs 30 which are employed as the objects are differentfrom each other, a buffer or the like for controlling the speed isrequired to be provided in the expander 20. Not only the read/write databut also commands to the HDDs 30 may be subjected toseparation/integration and multiplex transfer in a process of thepresent second embodiment.

Also, not only the duplex transfer employing the combination of thephysical link rate of 3.0 Gbps and 1.5 Gbps, but also, for example, 4×transfer to four HDDs 30 having the HDDs 30-side rate of 1.5 Gbps whenthe rate of the controller 10-side is 6.0 Gbps, can be performed in thesame manner. In this case, four physical ports are specified in thephysical port specifying areas in the particular command, as the objectof the 4× transfer.

The particular process in the second embodiment can be applied to allRAID types, particularly, can be applied to the RAID levels of {0, 3, 4,5}. When the process is applied to each of the RAID levels, therespective load on the controller is {medium, small, medium, medium}.The load is particularly small in RAID 3. When duplex transfer isperformed in the second embodiment with a slow rate (1.5 Gbps) of theHDDs 30-side, the performance ratio in comparison with the preceding art(configuration in which ALIGN is inserted) is 1.0 (the same performance)when simply compared with a case having fast HDDs (3.0 Gbps), and 2.0(two times of performance) when simply compared with a case having thesame speed of HDDs (1.5 Gbps). Similarly, when 4× transfer is performedin the second embodiment with a slow rate of the HDDs 30-side, theperformance ratio against the preceding art is 1.0 when compared withthe case having fast HDDs, and 4.0 when compared with the case havingthe same speed of HDDs.

Next, FIGS. 11A and 11B are explanatory diagrams representing a settingin a case in which the particular operation of the second embodiment isapplied in an actual RAID system. FIG. 11A shows a RAID group whichcorresponds to the particular operations of the second embodiment. FIG.11B shows a setting screen and a setting example of the RAID groups.

As shown in the upper side of FIG. 11A, in the disk array apparatus,first, a plurality, for example two, of the physical HDDs 30 are simplycombined, and a virtual HDD having a multiplied (doubled) capacity anddata-transfer speed is created (provided) by the physical HDD set(group). The number of HDDs in the physical HDD set which forms thevirtual HDD is determined by the system configuration such as physicallink rate ratio between the controller 10-side and the HDD 30-side. Inaccordance with needs, a plurality of virtual HDDs is created. Then, asshown in the low-speed side, a RAID group is formed over at least onecreated virtual HDD(s). For example, one RAID group is set over aplurality of virtual HDDs {#0 to #n}. The setting method employing thevirtual HDD can be applied to all the RAID levels. The processing formemploying the setting method described above is simple, therefore acircuit-addition is required only in the expander 20-side, and thereforea small-scale circuit can be realized. Meanwhile, no particular datamanipulation is required at the controller 10-side, and merely themanagement of the data position and the objective HDDs 30 are requiredto be performed by means of software.

In FIG. 11B, a user of the disk array apparatus performs setting of theRAID group, etc. by performing input operation on software which isprovided in the data processing device 300 or a maintenance device orthe like that are connected to the disk array apparatus. The upper sideshows an example in which physical HDD sets are formed by the pluralityof physical HDDs 30 which are connected to the expander 20. Thelow-speed-side shows an example in which virtual HDDs are formedcorresponding to the configuration of physical HDD sets. The settingprocedure of the RAID group, etc. is as the following. First, from theplurality of usable HDDs 30, the user selects a plurality of arbitraryHDDs 30 in accordance with, for example, the physical link rate ratio,and assigns them to virtual HDDs. For example, a set of physical HDDs #0and #1 forms one virtual HDD #0. Similarly, the procedure for forming avirtual HDD is repeated in accordance with needs to form a plurality ofvirtual HDDs. For example, arbitrary HDDs are employed from the physicalHDDs #0 to #11 and similarly aligned to the virtual HDDs, and sixvirtual HDDs #0 to #5 are formed in total. Next, already formed virtualHDDs are arbitrary grouped and a RAID group is set over them.Alternatively, arbitrary HDD among the already formed virtual HDDs isaligned to a spare (spare HDD). For example, five virtual HDDs #0 to #4are employed and a RAID group is formed over them, and one virtual HDD#5 is aligned to a spare HDD. Next, the RAID group set by the aboveprocedure is aligned to, for example, a logical unit (LU) or logicalvolume. For example, the RAID group over the virtual HDDs #0 to #4 isaligned to a logical unit with the RAID level set to RAID 5 and LUnumber set to LU0.

The disk array apparatus executes the particular operation of the secondembodiment by employing the RAID group created by the above describedsetting as the object. The process example of multiplex transfer to thedrive A and B shown in FIG. 10, corresponds to a process with onevirtual HDD. For example, when the controller 10-side physical link rateis 6.0 Gbps, four HDDs having the HDDs 30-side physical link rate of 1.5Gbps are grouped and a virtual HDD is formed. For example in a case inwhich the number of the HDDs for forming the RAID group is odd, the lastone HDD 30 which cannot form a physical HDD set for serving as thevirtual HDD is subjected to, for example, a processing mode in which theprocess is performed in a conventional method or alternatively, mixingwith dummy data is performed so as to form a virtual HDD. In this case,the effect is lower than the above described performance improvingeffects.

FIG. 12 is an explanatory diagram representing a process of a case inwhich the particular operation of the second embodiment is applied to adata copy operation performed in the disk array apparatus. Theparticular operation of the second embodiment can be employed in a datacopy operation (hereinafter, referred to as internal data copy)performed for data backup or the like in the disk array apparatuswithout mediation of the host.

When internal data copy is executed, the controller 10 reads multipleunits of data (for example A to D) which are the copy source data, fromthe HDD group (for example, HDD group A) having the copy source data,via the expander 20 by employing the multiplex transfer. Then, thecontroller 10 writes the read multiple units of data (A to D) to the HDDgroup (for example, HDD group B) for storing the copy destination data,via the expander 20 by employing the multiplex transfer. The copy datais subjected to the multiplex transfer via the controller-side physicallink 40, and distributed via the plurality of HDD-side physical links50. Accordingly, speed-up is realized in each of the read/writeoperations and also in the internal data copy.

The process employing the particular process of the second embodiment inthe internal data copy can be applied in the RAID level of {0, 1, 3, 4,5, 0+1, 3+1, 4+1, 5+1}. When the particular process is applied to eachof the RAID levels, the load on the controller 10 is small. When therate of the HDDs 30-side is slow and internal data copy is performed byemploying the duplex transfer in the second embodiment, the performanceratio against the preceding art is 1.0 when compared with a case havingfast HDDs, and 2.0 when compared with a case having the same speed ofHDDs.

Third Embodiment

Next, FIGS. 13A and 13B are the explanatory diagrams representing amodel of a particular process in a disk array apparatus of a thirdembodiment. FIG. 13A shows the flow of the process and the data betweenthe controller 10, the expander 20, and the HDDs 30. FIG. 13B shows therelation between input/output data and time in the controller 10 and theHDDs 30. The rate of the controller-side physical link 40 is 3.0 Gbps,and the rate of the HDD-side physical links 50 is 1.5 Gbps, thereforethe rate of one side is two times rate of the other side. The diagramshows a case in which data-write is performed on the HDDs 30 {drive Aand drive B} in the disk array apparatus, corresponding to a writeinstruction given from the host.

The third embodiment is based on the second embodiment, and, as aparticular operation, a plurality paths of data are subjected toaligning, rearrangement, or the like in advance in the controller10-side. Accordingly, formation and control of RAID is performed persets of physical HDDs 30. The multiplex transfer is performed with a setof arbitrary HDDs 30 in the same RAID group. The transfer data ismultiplexed and transferred via the controller-side physical link 40. Inthe multiplex transfer of the third embodiment, aligned data aretransmitted via the controller-side physical link 40. The expander 20performs distribution via the plurality of HDD-side physical links 50 inone-word units. In relation to the particular operation, the controller10 performs, as a data operation, data aligning corresponding to theRAID configuration, and the expander 20 performs, as a data operation,data distribution to the plurality (two) of HDDs 30 per word units. Thediagram particularly shows a case in which duplex transfer is performed.

In FIG. 13A, upon a write request, the controller 10 temporarily retainsthe write data {data A and data B} which have been given from the host,in the cache memory 15 without modification. The destination of write ofthe data A is the drive A, and the destination of write of the data B isthe drive B. The data sequences of the data A and B in word units arerespectively, {A0, A1, . . . , An} and {B0, B1, . . . , Bn}.

In respect to the plural (two) units of write data (A and B) which areemployed as the objects of the multiplex (2×) transfer, the controller10 sorts the data A and B, by the data separation/integration circuit17, based on word unit in accordance with the configuration of the RAIDgroup of the objective HDDs 30. The aligned data have a sequence of {A0,B0, A1, B1, . . . , An, Bn}. In the same manner as the embodimentsdescribed above, the controller 10 issues a particular command to theexpander 20 for instructing multiplex transfer. The controller 10sequentially transfers the particular command and the multiplex data tothe expander 20 via the fast-side physical link 40. The multiplex dataare the aligned data having a sequence of {A0, B0, A1, B1, . . . , An,Bn, . . . }.

The expander 20 receives the particular command and the multiplex datasequentially from the controller 10-side, and separates the multiplexdata by the data separation/integration circuit 27. The expander 20replicates the particular command, performs address conversion by use ofthe address table, and transmits the write command and the write datawhich have been separated into one-word units to each of the target HDDs30. The expander 20 transfers the write commands and the separated writedata to the HDDs 30 via the plurality (two) of slow-side physical links50. Upon the distribution at the expander 20, since the transfer datahave already been aligned in the controller 10-side, the data can besequentially transferred via the specified physical ports withoutmodification. The data A is transferred to the drive A, and the data Bis transferred to the drive B. Each of the HDDs 30 stores the separatedwrite data in the disk based on the received write command.

When the host reads data from the HDDs 30, the flow of the processdescribed above for write is reversed. That is, the expander 20 reads,the data in word units from the two HDDs 30 corresponding to the RAIDgroup which is the object of multiplex transfer, and integrates andtransfers the data to the controller 10-side. The controller 10 sortsthe data transferred from the expander 20 and provides the data to thehost.

As shown in FIG. 13B, in the relation between the input/output data andtime, for example, when the aligned data {A0, B0, A1, B1, . . . } aresequentially transmitted from the controller 10-side, the data {A0, A1,. . . } are correspondingly stored in the drive A at the same timing asthe transmission if the transfer delay time can be ignored. At thetiming delayed only by time t from the transmission process to the driveA, the data {B0, B1, . . . } can be correspondingly stored in the driveB.

In the third embodiment, the RAID configuration can be formed by a unitof one physical HDD 30 without employing the above described virtualHDD, and minute adjustments can be made. As compensation,circuit-addition is required also in the controller 10-side in terms ofhardware. The process of the third embodiment can be employed whendouble writing such as that of RAID 1 is performed, however, moreefficient method will be described in a sixth embodiment.

Next, a case in which the particular operation of the third embodimentis applied in an actual RAID system will be described. In this case, atthe beginning, the plurality of HDDs 30 is assigned so as to correspondto the RAID group, and the controller 10 executes multiplex transferwith a set formed by arbitrary two HDDs. This process is suitable formethods {RAID 4, 5, and 0} that handle a lot of small-size data.

FIG. 14A represents an example of processing procedure in a case inwhich the particular operation of the third embodiment is applied toRAID 5. Five HDDs #0 to #4 are provided as the HDDs 30 and are forming aRAID group. The HDD #4 is the position where parity is to be stored.First, a write request for the position of the HDD #0 is generated fromthe host-side (procedure 1). The controller 10 reads the correspondingdata from a set of HDDs #1 and #2 by employing the multiplex transfer(procedure 2), and subsequently, reads the corresponding data alone fromthe HDD #3 (procedure 3). Next, the controller 10 generates new parity(data P) by XOR operation of each of the data read from the HDDs #1 to#3 and the pre-write data to be written in the HDD #0 (procedure 4).Next, the controller 10 writes the pre-write data and the data P to aset of the HDDs #0 (data write position) and #4 (parity write position)by employing the multiplex transfer. The write of the data and theparity is completed by the above described procedure.

FIG. 14B represents a processing example in a case in which theparticular operation of the third embodiment is applied to RAID 0. FiveHDDs #0 to #4 are provided as the HDDs 30 and are forming a RAID group.First, a write request across the HDD #0 and #1 is generated from thehost-side (procedure 1). The controller 10 reads the corresponding datafrom a set of HDDs #1 and #2 by employing the multiplex transfer(procedure 2). Next, the controller 10 merges the data read from the HDD#0 and #1 and the data for write for the HDD #0 and #1 (procedure 3).Next, the controller 10 writes the merged data to a set of the HDD #0and #1 by employing the multiplex transfer (procedure 4). The write ofthe data is completed by the above described procedure. Meanwhile, whenthe data size of the write object is smaller than a stripe size, themultiplex transfer is not employed and direct write to each of the HDDs30 is performed.

FIG. 15 shows a setting screen for RAID groups corresponding to theparticular operation of the third embodiment. In the disk arrayapparatus, RAID groups are created by arbitrarily forming groups by theplurality of usable physical HDDs 30. A necessary number of RAID groupsare created. Then, assigning to LU or the like is performed with the setRAID groups. For example, a RAID group is formed by a set of HDDs #0 to#5, the RAID level thereof is set to RAID 5, and the LU number is set toLU0. Meanwhile, another RAID group is formed by a set of HDDs #6 to #11,the RAID level thereof is set to RAID 0, and the LU number is set toLU1.

In the third embodiment, when RAID control is performed based on theabove described settings, the multiplex transfer process which isexecuted in accordance with the RAID control is an automatic process inthe disk array apparatus, therefore, operations by the user is same asthat in a normal usage.

Fourth Embodiment

Next, FIGS. 16A and 16B are the explanatory diagrams representing amodel of a particular process in a disk array apparatus of a fourthembodiment. FIG. 16A shows the flow of the process and the data betweenthe controller 10, the expander 20, and the HDDs 30. FIG. 16B shows therelation between input/output data and time in the controller 10 and theHDDs 30. The xfer rate of the controller-side physical link 40 is 3.0Gbps, and the rate of the HDD-side physical links 50 is 1.5 Gbps, thatis, the rate of one side is two times rate of the other side. Thediagram shows a case in which data-write is performed on three HDDs 30{drive A, drive B, and drive C} in the disk array apparatus, inaccordance with a write instruction given from a host.

The fourth embodiment is an application of the second embodiment andmodification of the third embodiment, and is same as the thirdembodiment in that the configuration can be formed per one HDD 30 andminute adjustments can be made. In the fourth embodiment, as aparticular operation, multiplex transfer is performed with a set of atleast three HDDs 30 in a method specialized for RAID 3, and a parityprocess such as a parity insertion process is performed in predeterminedintervals although data aligning in the controller 10-side is notperformed. The transfer data including parity data is multiplexed andtransferred via the controller-side physical link 40. In the multiplextransfer of the fourth embodiment, the data with parity is sequentiallytransmitted via the controller-side physical link 40. The expander 20distributes the data in one-word units over the three or more HDD-sidephysical links 50. In relation to the particular operation, thecontroller 10 performs, as a data operation, a parity process (e.g.,generation/insertion of parity upon write, and verify/automatic datarecovery by use of parity data and removing parity upon read), and theexpander 20 performs, as a data manipulation, distribution of data overthe three or more HDDs 30 per word units. The diagram particularly showsa case in which 3× transfer is performed by employing “two HDDs forstoring data+one HDD for storing parity” as the objects, and the dataand the parity are distributed and stored in a set of three HDDs 30 thatare forming the slow-speed-side RAID group.

In FIG. 16A, upon a write request, the controller 10 temporarily retainsthe write data which has been given from the host, in the cache memory15 without modification. The write data is distributed and recorded inthe set of HDDs 30 which serves as the write destination, and the datais not particularly limited as the data that belonging to any of theHDDs 30.

In relation to the write data which are employed as the multiplexobject, the controller 10 performs parity generation/insertion processin accordance with the configuration of the objective RAID group throughthe process of the data separation/integration circuit 17. For example,the data sequence of the objective write data in word units is {A, B, C,D, . . . }. Corresponding to the RAID 3-control of the three HDDs 30,for example, the controller 10 performs calculation for generating andinserting parity to two words (data A and B) at an interval of one word(P0).

In the same manner as the above described embodiments, the controller 10issues a particular command to the expander 20 for instructing multiplextransfer. The controller 10 sequentially transfers the particularcommand and the multiplex data to the expander 20 via the fast-sidephysical link 40. The multiplex data is the data with parity, and forexample, have a sequence of {A, B, P0, C, D, P1, E, F, P2, . . . }.

The expander 20 receives the particular command and the multiplex datasequentially from the controller 10-side, and separates the multiplexdata by the data separation/integration circuit 27. The expander 20receives the particular command and the data with parity, which havebeen given from the controller 10-side via the physical ports, at thebuffer for controlling the speed. The expander 20 replicates theparticular command, performs address conversion by use of the addresstable, and transmits the write command and the write data which havebeen separated into one-word units, to each of the target HDDs 30. Theexpander 20 transfers the write command and the separated write data tothe HDDs 30 via the three slow-side physical links 50. Upon distributionat the expander 20, corresponding to the distribution, since thetransfer data have already been subjected to a parity process in thecontroller 10-side, the data can be transferred via the specifiedphysical ports without modification. For example, the data (non-paritydata) are transferred to the drives A and B which are for storing data,and the parity data are transferred to the drive C which is for storingparity, respectively. Each of the HDDs 30 stores the separated writedata in the disk based on the received write command.

When the host reads data from the HDDs 30, the flow of the process uponthe above described write is reversed. That is, the expander 20 readsthe data in word units from the three or more HDDs 30 that correspondingto the RAID group which is the object of multiplex transfer, andintegrates and transfers the data to the controller 10-side. Thecontroller 10 performs, with the data transferred from the expander 20,verify/automatic data recovery, a parity removing process, etc. by useof the parity data, and provides the data to the host.

For example, in an automatic data recovery process employing parity dataupon read, when failure data are present in an HDD 30 in the RAID group,the controller 10 recovers the data by performing an XOR operationprocess by use of the data in another HDD in the RAID group. Meanwhile,for example in an automatic data recovery process which employs paritydata and is performed toward a spare HDD, when failure data are presentin an HDD 30 in the RAID group, similarly, the controller 10 recoversthe data by employing other data in the RAID group, and writes therecovered data to the spare HDD via the expander 20.

As shown in FIG. 16B, in the relation between the input/output data andtime, when the above described data {A, B, P0, C, D, P1, . . . } aresequentially transmitted from the controller 10-side to the expander 20,the data {A, C, E, . . . } are correspondingly stored in the drive A atthe same timing if the transfer delay time can be ignored. At the timingdelayed from the process by the time t, the data {B, D, F, . . . } arecorrespondingly stored in the drive B. At the timing delayed by the timet in addition to that, the parity data {P0, P1, P2, . . . } arecorrespondingly stored in the drive C.

In the fourth embodiment, parity generation is performed in thecontroller 10-side, therefore the degree of freedom in selection of RAIDgroups, etc. is high. In addition, the circuit design of the expander 20is comparatively simplified. Besides, although the load on thecontroller 10 is reduced, the sum of the rates of the side of three ormore HDDs 30 becomes larger than the rate of the controller 10-side,therefore the above described buffer for controlling the speed isrequired to be provided. From the view point of the data transferefficiency, when it is configured such that the rate of the controller10-side and the sum of the rates of the HDDs 30-side have the same speedby, for example, multiplying the paths in the controller-side physicallink 40, the capacity of the buffer for controlling the speed isreduced, which is desirable. Also, 4× transfer, etc. can be performed inthe same manner when “three HDDs for storing data+one HDD for storingparity” are employed as the objects. Also, it may be configured suchthat an HDD corresponding to fast rate of 3.0 Gbps is mixed as the HDD30 (drive C) for storing the parity data.

Fifth Embodiment

Next, FIGS. 17A and 17B are explanatory diagrams showing a model ofparticular process at the disc array device in a fifth embodiment. FIG.17A shows a process and a flow of a data among a controller 10, anexpander 20, and a HDD 30. FIG. 17B shows a relation betweeninput/output data and time in the controller 10 and each HDD 30. A rateof a controller-side physical link 40 is 3.0 Gbps, and a rate of aHDD-side physical link 50 is 1.5 Gbps, and this is the case where aratio of rate is double. Further, FIG. 17B shows the case where, inresponse to a write instruction from a host, a write of the data forthree HDDs 30 {drives A, B, and C} are performed in the disc arraydevice.

The fifth embodiment is an application of the second embodiment and is amodification of the fourth embodiment, and is such that the parityprocess at the controller 10 side in the fourth embodiment is performedat the expander 20 side. In the fifth embodiment, as a particularoperation, not less than three HDDs 30 forming the same RAID group aremade into a set so as to perform a multiplex transfer, and the parityprocess is performed at the expander 20. Transfer data is multiplexedand transferred by the controller-side physical link 40. The multiplextransfer in the fifth embodiment is simply to send the data in order inthe controller-side physical link 40. Distribution is performed forone-word unit by the expander 20 in not less than three HDDs-sidephysical links 50. With respect to the particular operation, thecontroller 10 does not perform any special data operation, and theexpander 20 performs a parity process (parity generation and insertionand the like upon writing) as the data operation, and performs thedistribution of the data to not less than three HDDs 30. The presentdiagrams show the case where a duplex transfer is performed particularlyin the controller-side physical link 40, and the transfer is performedin parallel to three HDDs 30 by a HDD-side physical link 50, and threeslow-side HDDs 30 are made into a set so as to distribute the data andthe parity to be stored.

In FIG. 17A, at write request time, the controller 10 temporarily holdsa write data from a host in its state as it is in a cache memory 15. Thewrite data is distributed and registered in the set of the HDD 30, whichis a write destination, and the data of which HDD 30 it should be is notparticularly decided. The controller 10, with respect to the write datawhich is taken as a multiplex object, similarly to the above descriedembodiment, issues a particular command for instructing a multiplextransfer to the expander 20. The controller 10 transfers the particularcommand and the multiplex data in order to the expander 20 by thefast-side physical link 40. With respect to the multiplex data, the datasequence of one-word unit is shown as {A, B, C, D, . . . }.

The expander 20 receives the particular command and the multiplex datain order from the controller 10 in a buffer for controlling the speed,and performs separation of the multiplex data by a dataseparation/integration circuit 27. The expander 20 duplicates theparticular command, and performs an address conversion by an addresstable, and delivers a write command and the write data separated byone-word unit to each HDD 30 which becomes a target. At this time, withrespect to the write data, the expander 20 performs a parity generationand an insertion process corresponding to the formation of a RAID groupof the target by the data separation/integration circuit 27. Theexpander 20, for example, performs a calculation for forming andinserting the parity at the intervals of one word (P (A-B)), forexample, for two words (data A and B) corresponding to the control ofthe RAID 3 for three HDDs 30.

The expander 20 transfers the write command and the separated write datato the HDD 30 by three slow-side physical links 50. For example, thedata (non-parity data) is transferred to drives A and B for datastorage, and the parity data is transferred to the device C for theparity storage, respectively. Each HDD 30 stores the separated writedata in a disc based on a received write command.

At the read time of the data for the HDD from the host, the process flowis reversed with the flow upon writing. That is, the expander 20 readsthe data from three HDDs 30 corresponding to the RAID group which is themultiplex object by the word unit and subjects it to the parity process,and integrates and transfers it to the controller 10 side. Thecontroller 10 takes the transfer data from the expander 20 as a data forthe host. In the fifth embodiment, the expander 20 mainly performs anautomatic data restoration and the like at the read time.

As shown in FIG. 17B, with regard to the relation between input/outputand time, when the word unit data {A, B, C, D, . . . } is transmittedfrom the controller 10 side to the expander 20, the data is stored inthe drive A as corresponding data {A, C, E, G, . . . } at the sametiming as the transmission except for a transfer delay time. Thetransferred data is stored in the drive B as corresponding data {B, D,F, . . . } at the delayed timing of this process and time t. Further,the transferred data is stored in the drive C as corresponding paritydata {P(A-B), P(C-D), P(E-F), . . . } at the further delayed time t.

In the fifth embodiment, in the case of the number of HDD and aconfigurational example of the rate shown in FIG. 17, [the rate (3.0Gbps) of the controller side]=a total sum of the rates of the HDD (1.5Gbps+1.5 Gbps)]. In case of performing the mulplixing transfercorresponding to a RAID 3, since no particular data operation isperformed in the controller 10 side, a load of the controller 10 issmall. Further, when combined with the system (the third embodiment) forperforming the data operation in the controller 10 side, a RAID 4 can berealized.

Further, by addition of the parity distribution function to a pluralityof HDDs 30 in the expander 20 side or by addition of the function toissue the particular command while the designated order of the writeobject HDD 30 is changed by physical port designation every time theparity disc is changed by the controller 10 side, through the processbecomes complicated by that much, a RAID 5 can also be realized.However, the degree of freedom of setting regarding the number of HDDwithin the RAID group is not much high. With respect to this setting, amode of selecting and using a setting from several patterns according tothe configuration and the like is taken. Although a useable pattern islimited, since optimization is easy, it becomes fast.

With respect to the fifth embodiment, the case of performing the processcorresponding to the RAID 4 and the RAID 5 by the combination with thedata operation in the controller 10 side will be described. The casewhere HDD #0 to #3 of a slow rate (1.5 Gbps) are connected bycorresponding to four physical ports #0 to #3 carried by the expander 20will be taken as an example. Assuming that the data A, B, C areavailable as object data for the HDD 30. The sequence of the word unitof the data A, B, C is taken as {A0, A1, A2, . . . }, {B0, B1, B2, . . .} and {C0, C1, C2, . . . }, respectively.

The controller 10, at the write request time, transmits the particularcommand including a physical port number which is a transfer destinationto the expander 20 connected to the HDD group (HDD #0 to #3) of thetarget. That is, the controller 10 transmits the write command includingan expander address as the destination and the designation of thephysical port numbers #0 to #3 of the expander 20 as physical portinformation to the fast-side physical link 40. The controller 10 sortsand transfers the write data A, B, C for the HDD group of the targetsimilarly to {A0, B0, C0, A1, B1, C1, A2, B2, C2, . . . } to thephysical link 40. In this aligning, each data is alternately aligned bya striping unit (word unit).

The expander 20, with respect to the transfer data from the controller10 side, distributes the data (for example, A0, B0, and C0) to thephysical port for each HDD 30 according to the designated order of thephysical port in the accepted command. The expander 20 transfers theparity data (for example, P(A0-C0)) formed from the transfer data to theHDD 30 (parity disc) to be connected to the physical port designatedfinally by the accepted command.

At the time of transferring the data to the HDD 30 by the expander 20,provided that the physical port designated order which is a transferobject for every data (A, B, and C) of each system is not changed, andthat the physical port designated order is changed (shifted) for everystriping unit of the data of each system by the RAID 4 system, arecording by the RAID 5 system is made possible. As a process example inthe case of the RAID 5, at an initial timing, the data (A0, B0, C0 andP(A0-C0)) is distributed and written by corresponding to four HDDs 30(#0, #1, #2, #3). At the next timing, by designated shift of thephysical port, the data (P(A1-C1), A1, B1, and C1) is distributed andwritten by corresponding to the HDD (#0 to #3). Similarly, in the nexttiming, the data (C2, P(PA-C2), A2 and B2) is distributed and written.

According to the RAID control of the fifth embodiment, since the data isrequired to be written in all the HDD 30 forming the RAID group, theprocess such as making the striping size small and reading a data onceand overwriting a new data on it, and after that, writing back it withrespect to the portion where the remainder is left in the RAID group isrequired. Further, in case it is the RAID 3 system, a data sort is notrequired by the controller 10, which can be easily realized by makingthe physical repot designated order in the command constant.

Further, in the fifth embodiment, the parity is formed at the expander20 side, so that [the rate (3.0 Gbps) at the controller side<a sum (1.5Gbps×3=4.5 Gbps) of the rate at the HDD side]. Hence, even in case theHDD 30 of the same speed (3.0 Gpbs) as the rate of the controller sideis used as the HDD 30 (the HDD drives A to C) forming the RAID group,the effect of the upgrade of the performance can be obtained.

With respect to the fifth embodiment, an automatic data recovery usingthe parity at the read time will be described. FIG. 18 is an explanatorydiagram for the automatic data recovery using the parity at the readtime and a data recovery to a spare HDD and the like. At the data readtime, since the position of the parity disc (region in which the paritydata is stored) is defined clearly by a command to the expander 20 fromthe controller 10, even in a state of a trouble happening to one set ofHDD 30 in the RAID group, the data recovered within the expander 20 canbe delivered to the controller 10 as a read data. For this datarecovery, two types of usage method are available: the recovery of theread request data from the host and the recovery of a copy back data tothe HDD 30 such as a spare HDD, a replaced HDD and the like. The copyback is a process in which the data of the HDD 30 in a troubled statedue to breakdown and the like as well as the HDD30 as a replaced objectis moved to other HDD 30, and after that, the data is returned to theHDD 30 such as the spare HDD, the replaced HDD and the like. Thecontroller 10 executes only the read/write of the data, and therefore,its load is very small.

For example, the sequence of original data (object data to be read) istaken as {A0, B0, C0, A1, B1, C1, A2, B2, C2, . . . }. By the processupon writing, the data is distributed and stored similarly to {A0, B0,C0, P0} for the HDD 30 (#0 to #3) forming the RAID group. In the case ofthe RAID 4, the data A {A0, A1, A2, . . . } is stored in the HDD#0. Eachdata B, C, and P (parity data) are similarly stored in other HDD #1 to#3 also.

In the case of the read request from the host, the particular command(read command) is issued from the controller 10 to the expander 20. Theexpander 20 interprets the accepted command and reads the data inparallel from the HDD #0 to #3. At this time, for example, suppose thatthe HDD #1 is in a troubled state and the data B (B0, B1, B2, . . . ) isin error. The expander 20 uses the data of the HDD (#0, #2 and #3) whichare not in the troubled state, and performs a XOR operation, therebyrestoring the data B (B0, B1, B2, . . . }, which are, for example, ┌A0xor C0 xor P0=B0┘. The expander 20 puts together the restored data (dataB) and the read data to make it normal original data, and transfers itto the controller 10, and the controller 10 transmits it to the host asa response.

Further, in case of restoring a copy back data, the controller 10 readsthe data (data B {B0, B1, B2, . . . }) restored by the expander 20, andwrites it to one set of HDD #n such as the spare HDD, the replaced HDDand the like as a copy back data. Since the write destination of thecopy back data is one set of HDD #n, the write is performed by normalaccess. In this case, since the rate at the controller 10 side is largerthan the rate at the HDD 30 side, for example, issuing intervals of thecommand are adjusted at the controller 10 side so as to perform thetransfer process adapted to the performance of the HDD 30 side. Theadjustment at the controller 10 side makes it easy to divide the energyto spare for the host request process.

The particular process in the fifth embodiment is applicable to {3, 4,and 5} as a RAID level. The load of the controller 10 by the parityprocess upon writing becomes {small, medium, medium}. In case ofperforming a 4× transfer in the present embodiment where the rate of theHDD 30 is slow (1.5 Gbps) and [three HDDs for data storage+one HDD forparity storage] is taken as the object, a performance ratio compared toa conventional art is 0.75 compared to the case where the HDD is fast,and is about 2.67 compared to the case where the HDD is at the samespeed. Further, the load of the controller 10 in the parity process atthe read time becomes very small when adapted to each RAID level.Further, in case of performing the automatic data recovery by the parityat the read time by the 4× transfer, the performance ratio compared tothe conventional art is 3.0 compared to the case where the HDD is fast,and is 6.0 compared to the case where the HDD is fast. Further, in caseof performing the data recovery to the spare HDD (one set), a normalaccess is made, and the performance ratio compared to the conventionalart is 0.5 compared to the case where the HDD is fast, and is 1.0compared to the case where the HDD is at the same speed.

Sixth Embodiment

Next, FIGS. 19A and 19B are explanatory diagrams showing a model ofparticular process at the disc array device in a sixth embodiment. FIG.19A shows a process and a flow of a data among a controller 10, anexpander 20, and a HDD 30. FIG. 19B shows a relation betweeninput/output data and time in the controller 10 and each HDD 30. A rateof a controller-side physical link 40 is 3.0 Gbps, and a rate of aHDD-side physical link 50 is 1.5 Gbps, and this is the case where aratio of rate is double. Further, FIG. 19B shows the case where, inresponse to a write instruction from a host, a write of the data for twoHDD 30 (drives A and B) is performed in the disc array device.

In the sixth embodiment, as a particular operation, a plurality of HDDs30 (particularly two sets) are made into a set, and a multiplex writing(double writing) of the same data is performed on it, respectively. Bythe command issued once from the controller 10, the same data is writtenin a plurality of HDDs 30 by the expander 20. Not less than twoarbitrary HDDs 30 within the same RAID group are made into a set to bean object of the multiplex writing, and a data distribution for themultiplex writing is performed by a plurality of slow-side physicallinks 50. A plurality of HDDs 30, in which the rate of the physical linkconnected to the expander 20 becomes the same, are taken as themultiplex object. The multiplex writing is performed in a plurality ofHDDs-side physical links 40, for example, by one-word unit by theexpander 20. The multiplex transfer in the second embodiment is simplyto transmit the data in order in the controller-side physical link 40.With respect to the particular operation, the controller 10 does notperform any particular data operation, and the expander 20 performs thedata duplication for the multiplex writing to a plurality(two sets) ofHDD 30 as the data operation. The present diagrams particularly show thecase where two HDD 30 are made into a set to perform the double writing.

In FIG. 19A, at write request, the controller 10 temporarily holds awrite data from a host in its state as it is in a cache memory 15. Thecontroller 10, with respect to the write data taken as a double writingobject, issues a particular command for instructing the double writingto the expander 20. The controller 10 transfers the particular commandand the multiplex data in order to the expander 20 by the fast-sidephysical link 40. In the present diagrams, with respect to the doublewriting data, the data sequence of one-word unit is shown as {A, B, C,D, . . . }.

The expander 20 receives the particular command and the write data inorder from the controller 10 side in a buffer for controlling the speed,and performs a data separation for the double writing by a dataseparation/integration circuit 27. The expander 20 duplicates anaccepted command and a write data for a necessary volume. The expander20 performs an address conversion by an address table, and distributesthe write command and the write data of one-word unit to each HDD 30taken as the target by two slow-side physical links 50. For example, thecommand data and the write data {A, B, C, D, . . . } are transmitted inorder to the drives A and B, respectively. Each HDD 30 stores the writedata in the disc based on the received write data.

At the read time of the data for the HDD 30 from the host, the processflow is reversed with the flow upon writing. That is, the expander 20reads the data from two HDD 30 taken as the objects of the doublewriting, and transfers a normal read data to the controller 10 side. Thecontroller 10 takes the transfer data from the expander 20 as a data forthe host.

As shown in FIG. 19B, with regard to the relation between input/outputand time, when the data {A, B, C, D, . . . } of the word unit issequentially transmitted from the controller 10 side to the expander 20,the data is stored in the drives A and B respectively as thecorresponding data {A, B, C, D, . . . } at the same timing except for atransfer delay time.

In the sixth embodiment, since the RAID 1 system is automaticallyrealized and an overhead upon writing in the RAID 1 system can be madeapproximately null, the load of the controller 10 is lower than when thenormal RAID 1 system is formed. Since the controller 10 is only toexecute the read/write of the data, its load can be made very small.Further, [the rate (3.0 Gbps) of the controller side)=a total sum of therates (1.5 Gbps+1.5 Gbps) of the HDD side)], when compared by a dataunit, since the rate of the controller side becomes larger, a buffer forcontrolling the speed is required for the expander 20. Alternately, aprocessing method for inserting an ALIGN primitive in the controller 10side for adjusting the rate may be adopted. In this case, though thebuffer provided in the expander 20 can be made small, a data transferefficiency of the controller 10 side is lowered. Further, the HDD 30 ofthe fast rate (3.0 Gbps) can be used as the HDD 30 of the multiplexwriting object by corresponding to the controller 10 side. In case thecontroller side 10 and the HDD 30 side become the same speed, the bufferprovided in the expander 20 can be made the smallest minimum.

Further, since two HDD 30 taken as the objects of the double writingbecome the discs of totally identical value, in the case where whichside of the disc is involved is to be reliably determined when aredundant code (check code) for address check for every sector of thedisc is attached, the following method is applicable. First, there is amethod for mixing the data for each disc by using the process of thesecond embodiment. Alternatively, there is a method for generating acheck code for every disc of the double writing object in the expander20 side and inserting it into the data of the disc at constantintervals.

With respect to the sixth embodiment, the automatic data recovery usingthe double writing upon reading will be described. FIG. 20 is anexplanatory diagram for the automatic data recovery and the datarecovery to the spare HDD and the like by using the double writing uponreading.

At the data read time, even when the one HDD 30 in the double writtenHDD 30 is in a troubled state, the data of the response to thecontroller 10 can be transferred by the data of the other HDD 30. Sincean access is gained to two HDDs 30 by the expander 20, there is no needto re-execute an access to a mirror HDD in the controller 10 side, andthis results in an excellent efficiency. Further, a check by comparisonof the data from two HDDs 30 in the expander 20 side is also possible.For this data recovery, two types of usage method are available: therecovery of the read request data from the host and the recovery of acopy back data to the HDD 30 such as the spare HDD, the replaced HDD andthe like.

For example, the original data (read object data) is taken as A {A0, A1,A2, A3, A4, . . . }. By the double writing process upon writing, thedata A is stored in two HDD 30 (#0 and #1) forming the mirror,respectively.

Upon receipt of the read request from the host, the particular command(read command) is transmitted from the controller 10 to the expander 20.The expander 20 interprets the accepted command, and reads the data inparallel from the HDD #0 and #1. At this time, for example, suppose thatthe HDD #0 is in a troubled state, and its data A is in error. Theexpander 20 takes the read data from the other HDD#1 not in a troubledstate as the recovered data as it is. The expander 20 transfers therecovered data to the controller 10, and the controller 10 transmits itto the host as a response.

Further, in case of recovering the copy back data, the controller 10reads the data (data A) recovered by the expander 20, and writes it toone HDD #n such as the spare HDD, the replaced HDD and the like as thecopy back data. Since the write destination of the copy back data is oneHDD #n, the write is made by the normal access. Similarly to the case ofthe fifth embodiment, the transfer process is performed so as to beadapted to the performance of the HDD 30 with adjustment made at thecontroller 10 side.

The particular process in the sixth embodiment is applicable by the RAID1 as a RAID level. The load of the controller 10 in the replicatingprocess upon writing becomes very small. In case the rate of the HDD 30in the sixth embodiment is slow (1.5 Gbps) and the double writing isperformed with two HDD 30 as objects, the performance ratio compared tothe conventional art is 1.0 compared to the case where the HDD is fast,and is 2.0 compared to the case where the HDD is at the same speed.Further, in case of performing the automatic data recovery by the mirrorHDD, the load of the controller 10 is the same as the normal access, andthe performance ratio compared to the conventional art is 0.5 comparedto the case where the HDD is fast, and is 1.0 compared to the case wherethe HDD is at the same speed.

Seventh Embodiment

Next, FIGS. 21A and 21B are explanatory diagrams showing a model ofparticular process at the disc array device in a seventh embodiment.FIG. 21A shows a process and a flow of a data among a controller 10, anexpander 20, and a HDD 30. FIG. 21B shows a relation betweeninput/output data and time in the controller 10 and each HDD 30. A rateof a controller-side physical link 40 is 3.0 Gbps, and a rate of aHDD-side physical link 50 is 1.5 Gbps, and this is the case where aratio of rate is double. Further, FIG. 21B shows the case where, inresponse to a write instruction from a host, a write of the data forfour HDD 30 (drives A, B, C, D) is performed in the disc array device.

The seventh embodiment is an embodiment combining the sixth embodimentand the second embodiment, and has the features of the respectiveembodiments. In the seventh embodiment, as a particular operation, aplurality of HDDs 30 are made into a set so as to separate anddistribute the date, thereby performing a multiplex transfer, and at thesame time, with respect to this data distributed by the multiplextransfer, a plurality (particularly two sets) of HDD 30 are made into aset so as to perform a multiplex writing. The multiplex writing in theseventh embodiment is simply to transmit the data in order in thecontroller-side physical link 40. With respect to the particularoperation, the controller 10 does not perform any particular dataoperation, and the expander 20 performs the duplication of data for themultiplex writing as a data operation and the distribution of the databy a word unit to a plurality of HDDs 30. The present diagramsparticularly show the case where two HDDs 30 are made into a set toperform a double writing, and further, a duplex transfer is performedfor a pair of two sets each of the HDD 30.

In FIG. 21A, at write request time, the controller 10 temporarily holdsa write data from a host in its state as it is in a cache memory 15. Thecontroller 10 issues a particular command to the expander 20 withrespect to the write data taken as a particular operation object. Thecontroller 10 transfers the particular command and the write data inorder to the expander 20 by the fast-side physical link 40. The presentdiagram shows the process object write data as a data sequence ofone-word unit {A, B, C, D}.

The expander 20 receives the particular command and the write data in abuffer for controlling the speed in order from the controller 10 side,and performs a data separation corresponding to a double writing and amultiplex transfer by a data separation/integration circuit 27. Theexpander 20 duplicates an accepted command and the write date for anecessary volume. The expander 20 performs an address conversion by anaddress table, and transmits a write command and a write data byone-word unit to each HDD 30 taken as a target by four slow-sidephysical links 50. For example, a command data and write data {A, C, E,G, . . . } are transmitted in order to the drives A and B, respectively,and the command data and write data {B, D, F, . . . } are transmitted inorder to the drives C and D, respectively. Each HDD 30 stores thereceived write data in a disc based on a received write command.

At the read time of the data for the HDD 30 from the host, the processflow is reversed with the flow upon writing. That is, the expander 20reads the data from four HDDs 30 taken as the objects of the doublewriting and the duplex transfer, and integrates a original data andtransfers it to the controller 10 sides as a multiplex data. Thecontroller 10 takes the transfer data from the expander 20 as a data forthe host.

As shown in FIG. 21B, with regard to the relation between input/outputand time, when the data {A, B, C, D, . . . } of the word unit issequentially transmitted from the controller 10 side to the expander 20,the transferred data is stored in the drives A and B as correspondingdata {A, C, E, G, . . . } respectively at the same timing except for atransfer delay time, and is stored in the drives C and D ascorresponding data {B, D, F . . . } respectively at the delayed timingof the process for the drives A and B and the time.

With respect to the sixth embodiment, an automatic data recovery usingthe double writing upon reading will be described. FIG. 22 is anexplanatory diagram for the automatic data recovery and the datarecovery to a spare HDD and the like upon reading.

At the data read time, even when the one HDD 30 in the double writtenHDD 30 is in a troubled state, the data of the response to thecontroller 10 can be transferred by the data of the other HDD 30. Sincean access is gained to two HDD 30 by the expander 20, there is no needto re-execute an access to a mirror HDD (mirror disc) in the controller10 side, and this results in an excellent efficiency. Further, a checkby comparison of the data from two HDD 30 in the expander 20 side isalso possible. For this data recovery, two types of usage method areavailable: the recovery of the read request data from the host and therecovery of a copy back data to the HDD 30 such as a spare HDD, areplaced HDD and the like.

For example, an original data (read object data) is taken as the datasequence {A0, B0, A1, B1, A2, B2, . . . } of one-word unit which is amultiplex data of the data A and B. By the duplex transfer and thedouble writing process upon writing, the data A is stored in the mirrorHDD (#0 and #1), and the data B is stored in the mirror HDD (#2 and #3)for four HDD 30 (#0 to #3) forming the mirror HDD of two types.

In the case of the read request from the host, the particular command(read command) is transmitted from the controller 10 to the expander 20.The expander 20 interprets the accepted command, and reads the data inparallel from the HDD (#0 to #3). At this time, suppose that the HDD #2is in a troubled state, and the data B is in error. The expander 20takes the read data from the other HDD #3, which is not in a troubledstate with respect to the data B, as a recovered data as it is. Theexpander 20 transfers the recovered data to the controller 10, and thecontroller 10 transmits it to the host as a response.

Further, in case of recovering the copy back data, the controller 10reads the data (data B) recovered by the expander 20, and writes it toone HDD #n such as the spare HDD and the replaced HDD and the like.Since the write destination of the copy back data is one HDD #n, thewrite is made by the normal access. Similarly to the case of the fifthembodiment, the transfer process is performed so as to be adapted to theperformance of the HDD 30 with the adjustment made at the controller 10side.

The particular process of the seventh embodiment is applicable to {0+1,3+1, 4+1, and 5+1} as a RAID level. The load of the controller 10 in theduplex transfer and the replicating process upon writing when applied toeach RAID level becomes small. In case the duplex transfer and thedouble writing are performed with the rate of the HDD 30 being slow (1.5Gbps) and four HDD 30 taken as objects in the present embodiment, aperformance ratio compared to a conventional art is 2.0 compared to thecase where the HDD is fast, and is 4.0 compared to the case where theHDD is at the same speed. Further, in case of performing the automaticdata recovery at the read time, the load of the controller 10 becomesthe smallest, and a performance ratio compared to a conventional art is1.0 compared to the case where the HDD is fast, and is 2.0 compared tothe case where the HDD is at the same speed. Further, in case ofperforming the data recovery to the spare HDD (one set), the read andthe write of the recovered data are identically with the normal access,and the load of the controller 10 at each RAID level becomes small, andthe performance ratio compared to the conventional art is 0.5 comparedto the case where the HDD is fast, and is 2.0 compared to the case wherethe HDD is at the same speed.

Eighth Embodiment

Next, FIGS. 23A and 23B are explanatory diagrams showing a model ofparticular process at the disc array device in a fifth embodiment. FIG.23A shows a process and a flow of a data among a controller 10, anexpander 20, and a HDD 30. FIG. 23B shows a relation betweeninput/output data and time in the controller 10 and each HDD 30. A rateof a controller-side physical link 40 is 3.0 Gbps, and a rate of aHDD-side physical link 50 is 1.5 Gbps, and this is the case where aratio of rate is double. Further, FIG. 23B shows the case where, inresponse to a write instruction from a host, a write of the data for sixHDDs 30 (drives A, B, C, D, E and F) is performed in the disc arraydevice.

The eight embodiment is an embodiment combining the fifth embodiment andthe sixth embodiment, and has the features of the respectiveembodiments. In the eighth embodiment, as a particular operation, with aplurality of HDDs 30 as the objects, a multiplex writing (particularly,double writing) shown in the sixth embodiment is performed together witha parity process shown in the fifth embodiment by the expander 20. Thatis, a multiplex transfer is performed with not less than six HDD 30taken as objects where a physical link rate forming the same RAID groupis the same, and a data and a parity are multiplex-written,respectively. The separation and distribution of a plurality (forexample, three) of the data including the parity process are performedby the expander 20 by the slow-side physical link 50 regarding thetransfer data, and at the same time, and the multiplex writing(particularly, double writing) is performed on each data distributed bya command issued once from the controller 10. The transfer data ismultiplex-transferred by the controller-side physical link 40, and thedistribution of the data in a plurality of the HDD-side physical links50 is performed by the expander 20, for example, by one-word unit. Themultiplex transfer in the eighth embodiment is simply to transmit thedata in order in the controller-side physical link 40. With respect tothe particular operation, the controller 10 does not perform anyparticular data operation, and the expander 20 performs the distributionof the data, the parity process and a data duplication as a dataoperation, and performs the distribution of the data to a total sum ofnot less than six HDDs. The present diagrams particularly show the casewhere a 3× transfer (duplex transfer+parity process) for dividing thetransfer data into two portions by one-word unit by the HDD-sidephysical link 50 and further distributing them into three portions puttogether with the insertion of the parity data and the double writing ofeach distributed data are combined, and the data is distributed andstored by making a total of six slow-side HDD 30 into a set.

In the eighth embodiment, since the physical port used for theparticular operation is required not less than six, to designate, forexample, six physical ports in the physical port designated region of aparticular command shown in FIG. 7, the designation is performed, forexample, by using the following format. The controller 10 designates amode to perform the [3× transfer+double writing] process by theparticular command, and at the same time, by using the region of fourphysical port information (physical port No.) in the physical portdesignated region, designates three physical ports by the physical portnumbers from among six physical ports taken as usage objects by theparticular operation. The format is such that, by designating onephysical port number, the next physical port number is alsoautomatically designated. For example, in the case where six physicalports #0 to #5 are desired to be designated as the objects bycorresponding to six drives A to F in the expander 20, the physicalports #0, #2 and #4 corresponding to three drives A, C and E aredesignated by the particular command. By the designation of the physicalport #0, the drive B corresponding to the next physical port #1 isautomatically designated. Other formats may be such as designating thenumber of the physical port group already set up or designating aconsecutive physical port range (for example, the physical port #0 to#5) by two physical port numbers.

In FIG. 23A, at the write request, the controller 10 temporarily holds awrite data from the host in its state as it is in a cache memory 15. Thecontroller 10 issues the particular command corresponding to thedesignation of the process to the expander 20 with respect to the writedata of the process object. The controller 10 transfers the particularcommand and the write data in order to the expander 20 by a fast-sidephysical link 40. The present diagram shows the write data of theprocess object as a data sequence of one-word unit {A, B, C, and D}.

The expander 20 receives the particular command and the write data in abuffer for controlling the speed in order from the controller 10 side,and performs the distribution of the data with a total of six data as aunit by separation into three data with two data and one parity datataken as a unit and by duplication of the data for the double writing ofeach of those data by a data separation/integration circuit 27. In theparity process, for example, a parity data P1=P (A−B) of one word isgenerated from the data A and B of two words by a XOR operation. Theexpander 20 duplicates an accepted command and the write data for anecessary amount. The expander 20 performs an address conversion by anaddress table, and delivers a write command and the write data ofone-word unit to each HDD 30 taken as the objects by six slow-sidephysical links 50. For example, the command and the write data {A, C, .. . } are transmitted to the drives A and B, and the command and thewrite data {B, D, . . . } are transmitted to the drives C and D, and thecommand and the write data {P1, P2, . . . } are transmitted to thedrives E and F, respectively. Each HDD 30 stores the write data in thedisc based on the received write command.

At the read time of the data for the HDD 30 from the host, the processflow is reversed with the flow upon writing. That is, the expander 20reads the data from six HDD 30 taken as the process objects, andsubjects them to the parity process so as to integrate and transfer theoriginal read data to the controller 10 side as a multiplex data. Thecontroller 10 takes the transfer data from the expander 20 as a data forthe host.

As shown in FIG. 23B, with regard to the relation between input/outputdata and time, when the data sequence {A, B, C, D, . . . }is transmittedin order from the controller 10 side to the expander 20 is stored in thedrives A and B as the corresponding data {A, C, E, G, . . . } at thesame timing for the data transmission except for a transfer delay time.The transferred data is stored in the drives C and D respectively, asthe corresponding data {B, D, F, . . . } at the delayed timing of thetransfer process to the drives A and B with a delay time t. Further, thedata is stored in the drives E and F as the parity data {P(A-B), P(C-D),P(E-F), . . . } at the delayed timing of those process and further delaytime t. With respect to the HDD 30 (drives E and F) storing the parity,the HDD 30 corresponding to the fast rate (3.0 Gbps) can be used byadapting to the controller-side physical link 40.

With respect to the eighth embodiment, an automatic data recovery uponreading will be described. FIGS. 24A and 24B are explanatory diagramsfor the automatic data recovery upon reading and a data recovery to aspare HDD and the like.

At the data read time, even when two HDDs 30 among a RADI group are in atroubled state and a data read is in error, the date transfer to thecontroller 10 side is possible. At the controller 10 side, there is noneed to process the data recovery by re-access to a mirror HDD and theparity, and this results in an excellent efficiency. For this datarecovery, two types of usage method are available: the recovery of theread request data from the host and the recovery of a copy back data tothe HDDs 30 such as the spare HDD, a replaced HDD and the like.

For example, an original data (read object data) is taken as the datasequence {A0, B0, A1, B1, A2, B2, . . . } of one-word unit which is amultiplex data of the data A and B of two types. By a 3× transfer andthe double writing process upon writing, the data A is stored in themirror HDD (#0 and #1), and the data B is stored in the mirror HDD (#2and #3), and a parity data P is stored in the HDD (#4 and #5) for sixHDD 30 (#0 to #5) forming the mirror HDD of three paths.

As shown in FIG. 24A, in the case of the read request from the host,similarly to the seventh embodiment, the expander 20 reads the data inparallel from the HDD (#0 to #5) based on the particular command. Atthis time, suppose that two HDDs which are double written, for example,the HDD (#2 and #3) are in a troubled state and its data B{B0, B1, B2, .. . } is in error. In this case, since the expander 20 is unable toperform the data recovery by using the mirror data with respect to thedata B, it recovers the data B by a XOR operation by using the data (Aand P) of other HDD 30 (#0 and #4) among the RAID group. The expander 20transfers the normal read data adapted to the recovered data to thecontroller 10, and the controller 10 transfers it to the host as aresponse.

Further, in case of recovering the copy back data, the controller 10reads the data (data B) recovered by the expander 20, and writes thesame data as the copy back data to two HDDs (#m and #n). Since the writedestination of the copy back data is two HDD 30, the write is made bythe particular operation similarly to the seventh embodiment. Similarlyto the case of the fifth embodiment, the transfer process is performedso as to be adapted to the performance of the HDD 30 with the adjustmentmade at the controller 10 side.

Further, as shown in FIG. 24B, at the data read time from the HDD (#0 to#5), suppose that not two HDD 30 which are double written, but two HDDs30 storing a different data, for example, HDD (#1 and #2) are in atroubled state and each of the data A and B is in error. In this case,the expander 20 reads the data of each of the mirror HDD with respect tothe data A and B as it is, so that the data recovery is made possible.That is, the read data (A and B) from the HDD (#0 and #3) is taken asthe recovery data. The expander 20 transfers the normal read data A andB to the controller 10, and the controller 10 transmits the data to thehost as a response. Further, in the case of the recovery of the copyback data, the controller 10 reads the data (data A and B) recovered bythe expander 20, and writes a different data to two HDD (#m and #n) suchas the spare HDD, the replaced HDD and the like as the copy back data.Since the destination of the copy back data is two HDD 30, similarly tothe second embodiment, the write is made by the particular operation.Similarly to the case of the fifth embodiment, the transfer process isperformed so as to be adapted to the performance of the HDD 30 side withthe adjustment made at the controller 10 side. Further, even in case theparity data P is in error, similarly to the above described process, therecovery of the data is made possible by using other HDD 30 within theRAID group.

Ninth Embodiment

Next, FIGS. 25A and 25B are explanatory diagrams showing a model of aprocess by a HDD information reporting function comprised by a discarray device in a ninth embodiment. FIG. 25A is an explanatory diagramshowing an example of the HDD information report among a controller 10,an expander 20, and HDD 30. FIG. 25B shows a designated example of aphysical port by a special command in the HDD information reportprocess.

In the ninth embodiment, to effectively realize the function carried byeach of the above-described embodiment, a function (HDD informationreporting function) to report HDD information regarding the HDD 30 undercommand of the expander 20 is provided in addition to the configurationand the function of each of the above described embodiment. The HDDinformation reporting function conducts research on a HDD stateincluding a connecting state of the HDD 30 (presence or absence of theconnection) and a transfer rate of the already connected HDD (physicallink rate in the HDD-side physical link 50), and reports to thecontroller 10. When executing each of particular operations by thisfunction, research is conducted whether or not the physical link of thedesignated physical port is usable or suitable for an objectiveoperation. The recognition of the HDD information is made by a slow-sidephysical link 50 mainly by the expander 20. The recognized HDDinformation is reported to the controller side 10.

Each physical port is given a physical port number as unique recognitioninformation within the system. According to the process conforming to aSAS protocol, when the disc array device is activated, mutualconnections are established between each device, and by exchanging theID of each device, the number and type of connected devices aredetermined. Even in case the connection and disconnection of the devicetakes place during the operation, the event is reported. The controller10 and the expander 20 grasp a system configuration including the HDDstate by the process including a rate negotiation conforming to the SASprotocol and the HDD information report process, and select and executeeach of the particular operations by adapting to the systemconfiguration.

First, the HDD information report process by the HDD informationreporting function will be described. The process is executed accordingto the procedures (1) to (6) shown below.

Procedure (1): First, when the disc array device is activated (power ontime), a data transfer rate (physical link rate) for every physical portof the HDD 30 side is recognized as a normal operation by the ratenegotiation with the expander 20 and the HDD 30. This operation is anoperation according to the conventional configuration. For example, asshown in FIG. 25A, in the HDD-side physical link 50, the HDD 30 (#A and#B) of 1.5 Gbps are in a connected state for the physical port #1 and#2, and the HDD 30 (#c) of 3.0 Gpbs is in a connected state for thephysical port #3, and no connected state is recognized for the physicalport #4.

Procedure (2): The expander 20 collects the SAS address of each HDD 30connected to the device itself and the expander address of otherconnected expanders 20, and prepares an address table for a routing ofthe mutual connection of each portion within the device itself. Thisoperation is also according to the conventional configuration. In theaddress table, for example, the addresses of the HDD 30 (#A to #C) aremapped for the physical port (#1 to #3) carried by the expander 20.

Procedure (3): The controller 10, when activated, executes a ratenegotiation with the expander 20 regarding the controller-side physicallink 40. This operation is also according to the conventionalconfiguration. For example, the rate of the controller-side physicallink 40 is recognized as 3.0 Gbps.

Procedure (4): The controller 10 requests the expander 20 to a report onthe HDD information. The expander 20 reports on the HDD state recognizedby the rate negotiation in compliance with the request from thecontroller 10. By this report, the controller 10 recognizes the state ofeach HDD-side physical link 50. This report process may be executed by aprivate MIB (Management Information Base) by using a SMP (SerialManagement Protocol) or may be executed by the particular commandissuing function.

Procedure (5): The controller 10 executes a discovery of the HDD 30 withthe already connected HDD 30. This process may be executed with the HDD30 corresponding to the physical port reported to be in a connectedstate taken as an object provided that the expander 20 corresponds tothe report function shown in the procedure (4).

By the above described procedures, the controller 10 and the expander 20recognize the HDD information. The controller 10 and the user, based onthe recognition of the HDD information, taking into consideration also aratio of rates with the controller-side physical link 40 and theHDD-side physical link 50, decide the particular operation taken as anexecuting object and its attribute of the process, and perform asetting, a command execution and the like. For example, the controller10 recognizes by the HDD information reporting process that the rates ofthe HDD#A and #B are 1.5 Gbps, and taking into consideration that therate of the controller-side physical link is 3.0 Gbps and a ratio ofrate is double, performs a setting and an execution of the particularcommand so that the particular operation of the duplex transfer and thelike is executed with this pair of HDD 30 taken as an object.

In the procedure (4), for example, in case of performing the HDDinformation reporting process by issuance of the particular command fromthe controller 10, as shown in FIG. 25B, by using bytes 8 and 9 in theSAS address region within a header of the particular command, thedesignating of the physical port information and the reporting theretoof the HDD information are performed. In the HDD state shown in FIG.25A, the controller 10 designates the physical port number and the likewhich become check objects to the expander 20 by using the SAS addressregion at the request of the HDD information reporting. The controller10, for example, designates the physical port #1 and #2 to make anenquiry as to whether or not the rate in this pair is slow (1.5 Gbps)and is usable by the particular operation. The expander 20, incompliance with the designation of the physical port from the controller10 side, reports the corresponding HDD state. For example, the expander20 reports that the physical port #1 and #2 are slow (1.5 Gbps) andusable.

Further, as shown in FIG. 25B, as for how to designate the physicalport, each bit in the region designating the physical port informationmay be let correspond to the physical port. The format in the diagram isthe case where one bit is let correspond to one physical portion, andthe physical port #31 to #16 are let correspond to 16 bit of the bytes8, and the physical port #15 to #0 are let correspond to 16 bit of 9bytes. In this case, 32 pieces of physical port can be reported by onecommand only at a time.

As the content of the report to the controller 10 from the expander 20,it may be only about the physical port connected to the slow (1.5 Gbps)HDD 30 capable of forming a RAID group for the particular operation, andmoreover, may be only about the physical port of the HDD connectionunit. Further, usable physical port needs not to be reported by the bitand the like, but the numerical value capable of recognizing a rate anda state of each physical port may be reported. In case the report ismade by the SMP, neither of the controller 10 nor the expander 20performs any operation on the header of the command. The report contentat this time can be arbitrary set.

Next, with respect to the ninth embodiment, a report process in the caseof the HDD in a troubled state by using the HDD information reportingfunction in the case of the HDD being in a trouble will be described.The response to various types of the command requests from thecontroller 10 must be performed within a definite period of time. Hence,in the case where the one HDD 30 is put into a error state due to atrouble in the HDD group taken as the object of the particularoperation, the operation becomes partially different depending on thepresence or absence of the automatic data recovery function (recoveryusing the parity and the mirror). By using information reportingfunction, the report of the information when the HDD is in a trouble isperformed from the expander 20 to the controller 10, and the operationcorresponding to each state is performed by the expander 20.

FIG. 26 is a table, wherein the advisability of the automatic datarecovery by the expander 20 for each data in relation to the combination(a to g) of two HDD 30 in a state of {drives A and B} and acorrespondence between the above described corresponding embodiment andthe operation executed by the expander 20 are shown. As the state of theHDD 30, there are [no response], [error report] and [normal]. The [noresponse] is a state where there is no response available from the HDD30 to the command, and the [error report] is a state where apredetermined error report is made from the HDD 30 as a response.

Being common to each of the above described embodiments, the datarecovery is not possible in five cases of a: [no response]-[noresponse], b:[no response]-[error report], c:[error report]-[errorreport], d:[normal]-[no response], and f:[normal]-[error report] in thecombination of the states of drives A-B. In the case of the fifth toeighth embodiments, the automatic data recovery may be possible in twocases of e: [normal]-[no response] and g:[normal]-[error report] in thecombination of the states of the drives A-B. Subsequently, according toeach case of a to g, the expander 20 performs the report and itsrelative operation for the controller 10 sides by using the HDDinformation reporting function.

In the case of a, the expander 20 reports no response error to thecontroller 10 after waiting until the threshold value of the processingtime. In the case of b, the expander 20 reports an error code of thedrive B and the no response error to the controller 10 after waitinguntil the threshold value of the processing time similarly to the caseof a. In the case of c, the expander reports the error codes of thedrives A and B to the controller 10. In the cases of a to c, theexpander 20, when reporting, mixes the error code (value showing theerror content) into the transfer data and transfers. The no responseerror information is generated by the expander 20.

In the case of d, the expander 20 reports the normal side data and theno response error after waiting until the threshold value of theprocessing time. At this time, the expander 20, when transferring, mixesthe data and the error code, and transfers the remainder by padding itby a dummy data. Further, the error report may be made such that thenormal side data is accessed by itself alone for the controller 20. Inthe case of e, the expander 20 reports the recovery data at the expander20 to the controller 10 after waiting until the threshold value of theprocessing time. Further, the expander 20 informs an intention of thedata recovery.

In the case of f, the expander 20 reports the normal side data and theerror code of the drive B to the controller 10. At this time, similarlyto the case of d, the expander 20 performs the mixing and the like ofthe data and the error code. In the case of g, the expander 20 reportsthe recovery data at the expander 20 to the controller 10, and moreover,reports the intention of the data recovery so as to hold the error codeat the expander 20. In the case of g, the report of the errorinformation at the time of the automatic data recovery may be made tothe controller 10. The controller 10, when informed of the intention ofthe data recovery from the expander 20, conducts research on the errorinformation (error code) to determine whether or not the error of theobject HDD 30 is serious or whether or not it is temporarily. In casethe automatic data recover is performed at the expander 20, since itbecomes a normal sequence, no report of the error information is made tothe controller 10. Hence, this error information is temporarily held ina memory at the expander 20 side, and the report (transmission) on theerror information held according to the occurrence of the request fromthe controller 10 is made.

As described above, in the ninth embodiment, by using the HDDinformation reporting function, the function carried by each of theabove described embodiments can be effectively performed. By the HDDinformation reporting function, the number of direct accesses to eachHDD 30 connected to the disc array device can be reduced, therebyimproving a traffic.

Tenth Embodiment

FIG. 27 shows an explanatory diagram representing a model for theparticular process in the disk array apparatus according to a tenthembodiment. It illustrates processes and data flows between thecontroller 10, expander 20 and HDD 30. The rate of the controller-sidephysical link 40 is 3.0 Gbps, and that of the HDD-side physical link 50is also 3.0 Gbps. The ratio of the rates is, therefore, one (1). Itshows a state in which data are written, in particular, for two HDDs 30(drives A and B) of the disk array apparatus in response to writinginstructions from the host.

The tenth embodiment provides data compression/decompression functionsbetween the controller 10 and the expander 20 in addition to thefunctions described at each embodiment. The tenth embodiment executessuch a particular operation that the controller-side physical link 40compresses transfer data for multiplex transfer for a plurality of HDDs30 and the a plurality of HDDs-side physical links 50 expands thetransfer data to distribute. The multiplex transfer in the tenthembodiment refers to the compression and transmission of data on pluralpaths at the controller-side physical link 40. With the particularoperation, the controller 10 compresses and expands a plurality of data,and the expander 20 compresses and expands a plurality of data as dataoperation and distributes data to a plurality of HDDs. The controller 10and the expander 20 further include data compression/decompressioncircuits in their respective data separation/integration circuits (17and 27). The figure shows the process of duplex transfer andcompression/decompression for a pair of two HDDs 30 with a high speedrate (3.0 Gbps).

In FIG. 27A, when received a request for writing, the controller 10publishes a particular command (write command) instructing doublingtransfer and compression/decompression processes of write data for twoHDDs 30 that is subjected to doubling process to the expander 20. Thecontroller 10 compresses user data and redundant code (check code) ofthe write data through the data compression/decompression circuit. Thecontroller 10 sequentially transfers the particular command andcompressed data to the expander 20 through the fast-side physical link40. For example, let the sequence of original data and that ofcompressed data be {A, A′, B, B′, C, C′, . . . } and {Ac, Bc, Cc, . . .} respectively. The data A and A′ lead to the data Ac by compression.The compression ratio by data compression/decompression process is letbe 50% as an example. The transfer process time is let be t incompressed data unit (Ac, and others). Where, commands are exempted fromcompression.

The expander 20 receives the particular command and compressed data insequence from the controller 10, expands the received data by the datacompression/decompression circuit, and distributes the data to aplurality of HDDs-side physical link 50. The expander 20 delivers writecommand and write data (decompression data) to each HDD 30 to be atarget. In the slow-side physical link 50, decompressed two of the datacorresponding to data of one compressed data unit according to rate andthe compression ratio requires a transfer-processing time of 2 t. EachHDD 30 stores the write data in disk in response to the write commandreceived.

When data are read from the instruction of the host, the flow is reverseto the writing flow. That is, the expander 20 reads data from two HDDs30 that are subjected to doubling process, compresses and integrateseach read data, and transfers the multiplexed data to the controller 10.The controller 10 expands the transfer data from the expander 20 to sendthem to the host.

FIG. 27B shows the relation between data and time. For example, when thesequence of the compressed data {Ac, Bc, Cc, . . . } are transmittedsequentially, the data from the controller 10 {A, A′, C, C′, E, E′, . .. } and {B, B′, D, D′, F, F′, . . . } are stored in the drives A and Brespectively with a delay of a decompressing process time about eachcompressed data at the expander 20.

In the tenth embodiment, even if the rate of the controller-sidephysical link 40 is equal (3.0 Gbps) to that of the HDDs-side physicallink 50, the effect of improvement in performances by the multiplextransfer can be obtained. For instance, a compression ratio is 50%, aswith the same process of the first embodiment, traffic can be halved atthe bus of the controller-side physical link 40. For instance, a userdata part in the data pattern is the same during format processing ofthe disk array apparatus, which produces a substantial compressioneffect caused by the data compression/decompression function. Inconsequence, the improvement of traffic at the controller 10 can reducea formatting time for the device.

Eleventh Embodiment

FIG. 28 shows a block diagram illustrating the configuration of the diskarray apparatus according to the eleventh embodiment. The disk arrayapparatus according to the eleventh embodiment has the basic chassis 120and the additional chassis 130, which include the controller 10, theexpander 20B, the data separation/integration end device 400, and HDD30. The figure shows the data flow at the host's request for write amongthe host 300, the controller 10, the expander 20B, the dataseparation/integration end device 400, and HDD 30.

In the eleventh embodiment, functions for controlling processesincluding particular operation in each embodiment described areimplemented not in the circuit of the expander 20 but in the dataseparation/integration end device 400 that is another end deviceconnected to the expander 20B. Thereby the same functions can beprovided. This figure shows the configuration in which, the additionalchassis 130 has the data separation/integration end device 400 havingfunctions equivalent to the data separation/integration circuit 27 isconnected outboard to the expander 20B without the dataseparation/integration circuit 27 through a bus and others. The host,the controller 10, HDD 30, and others are connected to one another as isthe case with the embodiment. The expander 20B conducts processes inresponse to SAS except for functions including the particular operationin the each embodiment described.

The data separation/integration end device 400 is a device such as LSIand others with software and hardware for actualizing functionsincluding the particular operation in the each embodiment. Providing anoption to connect or disconnect the data separation/integration enddevice 400 to the expander 20B actualizes scalability of the disk arrayapparatus.

When implementing the functions of the particular operation and others,the controller 10 specifies the SAS address to be set corresponding tothe data separation/integration end device 400 with the physical link 40between the expander 20B and the controller 10, publishes the particularcommand to transmit. Thereby, the controller 10 conducts data transferwith the data separation/integration end device 400.

The data separation/integration end device 400 has at least one or morepaths 401 for communication with the controller 10. The rate of the path401 is set to high speed (3.0 Gbps) to meet that of the controller 10and the expander 20B. Further, the data separation/integration enddevice 400 has a plurality of another paths 402 for communication withthe HDD 30. One of the paths 402 beside the HDD 30 may be shared withthe path 401 beside the controller 10 in the configuration.

The data separation/integration end device 400 has functions of formingand holding an address table by searching SAS address of HDD 30connected to the expander 20B connected corresponding the dataseparation/integration end device 400. The data separation/integrationend device 400 converts the SAS address (destination SAS address) in thecommand transmitted to the device itself by the expander 10 to the SASaddress of each HDD 30 to be targeted with the address table, and thenseparates and integrates the transfer data to transmit. The dataseparation/integration end device 400 converts the SAS address (sourceSAS address) of the controller 10 in the accepted command to the SASaddress of device itself. The data separation/integration process in thedata separation/integration end device 400 corresponds to various datamanipulations such as data distribution for multiplex transfer, datareplication for multiplex writing, and parity process described in theeach embodiment. When transferred from the HDD 30 to the controller 10,for example, on request for reading, data is transferred in reverse flowto the above mentioned in the same manner.

If necessary, the data separation/integration end device 400 may beconnected to inside the frame (400B) shown in the dotted line in theexpander 20B as well as connecting it outboard to the expander 20B.

A process flow of the disk array apparatus in one embodiment accordingto the present invention is described below. FIGS. 29 and 30 are flowcharts corresponding to processes in the disk array apparatus in eachembodiment stated above, and illustrating the process flow in the diskarray apparatus having functions in the each embodiment, especially,comprehensively, as one embodiment. In the device, the particularoperation is implemented selectively according to states of HDD andconnection rate, types of data to be processed, and setting related tothe implementation of the particular operation.

FIG. 29 shows a flow chart of the operation of the expander 20, in whichare illustrated the steps for transferring data (data writing andothers) to the HDD 30 based on the command from the controller 10.

On accepting frames from the controller 10 via the controller-sidephysical link 40 (step S101), the expander 20 refers to the header ofthe accepted frame to confirm whether an address is destined for thedevice itself (expander address) (S102).

If the address is not destined for the device itself (S102-NO), theexpander 20 delivers the frame to the physical port corresponding tospecified address within the frame (S103) and completes the process. Ifthe address is destined for the device itself (S102-YES), the expander20 refers to the flag area and command area on the header of the framefor checking (S104). The expander 20 determines whether a combination offlag and command is correct (S105) through the check. If the combinationis incorrect (S105-NO), the expander 20 conducts error processing andreporting to the controller 10 (S106) and ends the operation.

If the combination is correct (S105-YES), the expander 20 checks thearea specified as the physical port of the header in the frame (S107).The expander 20 recognizes the specified physical port subjected toprocessing such as data transfer through the check. Particularly, in theparticular operation, a plurality of physical ports will be thespecified physical port.

The expander 20 replicates the header of the accepted frame, or thecommand for each specified physical port through the dataseparation/integration circuits 27 (A108). An original command to betransmitted to a plurality of HDDs 30 subjected to processing is formedby the replication.

The expander 20 converts the address of a plurality of frames formed bythe replication at the area on the SAS address of the header (S109). Theaddress is converted by reference to the address table. The expander 20replaces the source address with the address destined for the deviceitself (expander address) in the area. At this point the reserve area iscleared. The expander 20 replaces the destination address with the HDDaddress corresponding to the specified physical port in the area.

The expander 20 determines whether the subject data (transfer data fromthe controller 10) is multiplexed data (S110). If the subject data is amultiplexed one, the data separation/integration circuits 27 separatesthe subject data for each the specified physical port (S111). Theseparation means a process according to types of the particularoperation. In duplex transfer such as the first embodiment and others,for example, data are separated into two physical ports.

The expander 20 determines whether it should form automatically the dataframe and parity of the subject data (S112). When the parity is formed,an XOR operation is executed based on the subject data to form it foruse in the specified physical port, as described in the fifth embodimentand others (S113).

The expander 20 determines whether the subject data should be writtendouble (multiplex writing) (S114). If a double writing is needed, theexpander 20 replicates the data to be written double for each specifiedport (S115).

The expander 20 publishes the converted and formed frame at each processmentioned above to the specified port (S116) and ends.

FIG. 30 is a flow chart showing the operation of the expander 20 withsteps for data transfer process (e.g., data reading) from HDD 30 basedon the command from the controller 10.

On receipt of a frame from the HDD 30 via the HDD-side physical link 50(S201), the expander 20 refers to the header of the frame to make sureif the destination address is the address destined for the device itself(expander address) (S202). If the address is not the address destinedfor the device itself (S202-NO), the expander 20 delivers the frame tothe physical port on the specified address in the frame (S203) and endsthe process.

If the address is the one destined for the device itself (S202-YES), theexpander 20 determines whether the HDD 30 that is the sender of theframe is in error (S204). If the HDD 30 is not in error (S204-No), theexpander 20 sorts the transferred data from the HDD30 on the memory(buffer) to integrate them (S205). If the HDD 30 is in error, theprocess is not executed.

The expander 20 determines whether the HDD 30 (including a HDD groupsubjected to the particular operation) has completed its response(S206). If the response has not ended (S206-NO), the expander 20determines whether the waiting time is in the permissible time (S207).If it is in the permissible time (S207-YES), the expander 20 ends theprocess. If the waiting time exceeds the permissible time (S207-NO) andthe HDD 30 subjected to the particular operation has already completedits response (S206-YES), the expander 20 determines whether all datafrom the HDD 30 are normal (S208).

If all data from the HDD 30 are normal (S208-YES), there is no need torecover the data. If the data include errors (S208-NO), the expander 20determines whether the data can be recovered using the parity, Mirror,and others (S209). If the data cannot be recovered (S209-NO), theexpander 20 conducts the error processing and reporting (S210), and endsthe process. If the data can be recovered (S209-YES), the expander 20executes a data recovery process (S211).

The expander 20 converts the address of the accepted frames in the areaon the SAS address of the header thereof (S212). The address isconverted with reference to the address table. The expander 20 replacesthe source address with the address of the device itself (expanderaddress). It also replaces the destination address with controlleraddress (SAS address of the controller 10). If there are the errorinformation and recovery information (for example, informationrepresenting data recovery by the expander 20) in the area, the expander20 sets them in the reserve area.

The expander 20 publishes the frames converted and formed at theaforementioned process to the physical port beside the controller 10(S213), and ends the process. The process flow terminates here.

In addition to the above, an ALIGN primitive may be inserted into forthe difference between longer and shorter data lengths if the transferdata length for a plurality of HDDs 30 are different. Alternatively, thedifference may be padded with a data dummy. With regard to theimplementation of the particular operation, it is allowable to implementthe multiplex transfer only during transferring data exerting a greatinfluence on traffic. It is also allowable to multiplex-transfer onlydata (user data). It is allowable to multiplex-transfer data along withall commands and status. According to the situation, it is allowable toselect data to be multiplexed.

As described above, in the disk array apparatus of the each embodimentaccording to the present invention, multiplexing the transfer data in aplurality of HDD-side physical links 50 as one set by thecontroller-side physical link 40 does not require any insertion of theALIGN primitive at the controller-side physical link 40 if there is adifference in rate between physical links in the connection of thecontroller 10 and HDD 30 and in data path. Thereby, the controller 10can deliver its full performances, transferring data effectively. Inaddition, it is possible not only for the controller 10 to deliver itsfull performances but for the bus to be used efficiently for theHDD-side physical link 50, thereby to improve performances byimprovement of the total traffic of the data transfer system.

The RAID system for storing distributed data in a plurality of HDDs 30needs to provide a plurality of data in the storing process. On theother hand, in the embodiment of the present invention, an overhead forproviding a plurality of data for data distribution at the slow-sidephysical link 50 is an inside content of the overhead at the particularoperation of the total device, which exerts little influence onperformances. Since the transfer data of each HDD 30 at the set of HDD30 subjected to multiplex transfer concentrates in the controller 10,the frequency and the time of bus occupancy are reduced, acceleratingdata flow, and improving traffic of the data transfer system.

Since an inexpensive but slow-speed HDD 30 (1.5 Gbps) may be used as astorage device without lowering data transfer performances, the cost ofthe total system can be reduced. The HDD 30 side need not be a type ofhigh speed (for example, 3.0 Gbps) for matching the rate of thecontroller 10, which provides an advantage of securing a technicalstability.

With the configuration and functions of the disk array apparatus andcommands described in each embodiment of the present invention, they areapplicable to either case in which the data transfer rate is differentor the same (nearly same) between the controller 10 and HDD 30 (SAS-HDD,and SATA-HDD) via the expander 20.

The present invention made by the inventor has been described above indetails based upon the embodiments. The present invention is not limitedto the embodiments, it is to be understood that the embodiments can bechanged without departing from the scope and spirit of the presentinvention.

The present invention can be applied to an SAS to be connected throughan SAS expander device and a disk array apparatus connected to a storagedevice corresponding to SATA, and a computer system.

1. A storage system comprising: a plurality of disk devices; an SASexpander coupled to the plurality of disk devices via first physicallinks having a first level transmission rate; and a controller coupledto the SAS expander via a second physical link having a second leveltransmission rate, wherein the first level transmission rate is lowerthan the second level transmission rate, wherein the controller obtainsinformation to determine the first level transmission rate and thesecond level transmission rate, determines a multiplexing ratio based onthe determined first and second level transmission rates, multiplexesdata to be stored in at least two disk devices of the plurality of diskdevices by using time division multiplexing according to the determinedmultiplexing ratio, transmits a predetermined parameter to the SASexpander to indicate that subsequent data are multiplexed, and transmitsthe multiplexed data to the SAS expander, and wherein when the SASexpander receives the predetermined parameter via the second physicallink, the SAS expander demultiplexes the multiplexed data received afterthe predetermined parameter, and transmits demultiplexed data to the atleast two disk devices in parallel to store the demultiplexed data inthe at least two disk devices.
 2. A storage system according to claim 1,wherein the second level transmission rate is n times higher than thefirst level transmission rate, wherein n is an integer larger than 1,and wherein the controller determines the multiplexing ratio as n.
 3. Astorage system according to claim 2, wherein the first leveltransmission rate is 3.0 Gbps, and the second level transmission rate is6.0 Gbps.
 4. A storage system according to claim 1, wherein thecontroller transmits a request to obtain the information to determinethe first level transmission rate, and wherein the SAS expanderrecognizes the first level transmission rate by rate negotiationoperations between the SAS expander and each of the plurality of diskdevices, and reports the information to determines the first leveltransmission rate to the controller in response to the request from thecontroller.
 5. A storage system according to claim 4, wherein thecontroller obtains the second level transmission rate by ratenegotiation operations between the controller and the SAS expander, andwherein the controller calculates the ratio of the second leveltransmission rate to the first level transmission rate to determine themultiplexing ratio.
 6. A storage system according to claim 1, whereinthe storage system further includes another disk device coupled to theSAS expander via a third physical link having a third level transmissionrate which is at the same level as the second level transmission rate.7. A storage system according to claim 6, wherein when the controllercontrols to write data in the another disk device, the controllertransmits the data to the SAS expander without multiplexing.
 8. Astorage system according to claim 1, wherein the controller multiplexesthe data by a unit of a dword.
 9. A storage system according to claim 1,wherein when the SAS expander receives the predetermined parameter fromthe controller, the SAS expander demultiplexes the subsequent data, andwhen the SAS expander does not receive the predetermined parameter fromthe controller, the SAS expander transfers data received from thecontroller to a destination disk device of the data withoutdemultiplexing.
 10. A storage system according to claim 1, wherein whenthe controller reads data from the at least two disk devices, data aretransmitted from the at least two disk devices to the SAS expander inparallel, and the SAS expander multiplexes data received from the atleast two disk devices and transmits the multiplexed data to thecontroller.
 11. A storage system according to claim 10, wherein thecontroller demultiplexes the data received from the SAS expander, andtransmits demultiplexed data to a computer coupled to the storagesystem.
 12. A storage system according to claim 10, when the controllerreads the data from the at least two disk devices, the controllertransmits the predetermined parameter to the SAS expander, and the SASexpander reads the data from the at least two disk devices in parallel,multiplexes the data received from the at least two disk devices andtransmits the multiplexed data to the controller according to thepredetermined parameter.
 13. A storage system according to claim 10,wherein the controller controls the SAS expander to receive data fromthe at least two disk devices in parallel, multiplex the received dataand transmits the multiplexed data to the controller based on thedetermined multiplexing ratio.
 14. A storage system according to claim10, wherein the SAS expander includes a circuit executing multiplexingand demultiplexing operations.
 15. A storage system according to claim14, wherein the SAS expander is configured to couple to a deviceincluding another circuit executing multiplexing and demultiplexingoperations.
 16. A storage system comprising: a plurality of diskdevices; an SAS expander, to which each disk device is coupled via afirst physical link having a first level transmission rate; and acontroller coupled to the SAS expander via a second physical link havinga second level transmission rate, wherein the first level transmissionrate is lower than the second level transmission rate, wherein thecontroller obtains information to determine a ratio of the second leveltransmission rate to the first level transmission rate, determines amultiplexing ratio based on the determined ratio, instructs the SASexpander to read out data from at least two disk devices of theplurality of disk devices and multiplex the data by using time divisionmultiplexing according to the determined multiplexing ratio to transmitmultiplexed data to the controller, wherein, according to an instructionfrom the controller, the SAS expander reads out the data from the atleast two disk devices, multiplexes the data by using time divisionmultiplexing according to the determined multiplexing ratio, andtransmits the multiplexed data to the controller, and wherein when thecontroller receives the multiplexed data from the SAS expander, thecontroller demultiplexes the multiplexed data and sends the data to ahost computer coupled to the controller.
 17. A storage system accordingto claim 16, wherein the second level transmission rate is n timeshigher than the first level transmission rate, n being an integergreater than one, and the controller determines the multiplexing ratioas n.
 18. A storage system according to claim 17, wherein the firstlevel transmission rate is 3.0 Gbps, and the second level transmissionrate is 6.0 Gbps.
 19. A storage system according to claim 16, whereinthe controller obtains information of the second level transmission rateby rate negotiation operations between the controller and the SASexpander, and obtains information of the first level transmission ratefrom the SAS expander, wherein the SAS expander obtains the informationof the first level transmission rate by rate negotiation operationsbetween the SAS expander and the plurality of disk devices and reportsthe information of the first level transmission rate to the controller,and wherein the controller calculates the ratio of the second leveltransmission rate to the first level transmission rate based on theobtained information of the first and second level transmission rates.20. A storage system according to claim 16, wherein the storage systemfurther includes at least one another disk device coupled to the SASexpander via a third physical link having a third level transmissionrate which is at the same level as the second level transmission rate,wherein when the controller reads data from the another disk device, thecontroller instructs the SAS expander not to multiplex data to betransmitted from the SAS expander to the controller based on a ratio ofthe second level transmission rate to the third level transmission rate,and wherein the SAS expander reads out the data from the another diskdevice and transmits the data to the controller without multiplexing.21. A storage system according to claim 16, wherein the SAS expandermultiplexes data by a unit of a dword.
 22. A storage system according toclaim 16, the ratio of the second level transmission rate to the firstlevel transmission rate is equal to the multiplexing ratio.